msapi://

Note

msapi:// is available only with the Enterprise license.

The scheme msapi:// provides access to Azure Active Directory, Office 365 and Azure activity audit logs through Azure Active Directory Activity reports, Office 365 Management Activity and Azure Monitor REST APIs.

The Azure Active Directory Activity reports API provides two types of activity reports: audit and sign-in logs. The directory audit report provides you with records of system activities for compliance. The sign-in reports helps you determine who performed the tasks reported by directory audits. There are several reports representing different type of sign-in operations: interactive and non-interactive user, service principal and managed identities sign-ins.

The Office 365 Management Activity API provides information about user, admin, system, and policy actions and activities. The events are organized into different logs by content type (AzureActiveDirectory, Exchange, SharePoint, DLP and General).

The Azure Monitor REST API provides Azure activity logs regarding operations on each Azure resource in the subscription of a tenant, as well as events generated on the tenant level.

Microsoft provides the data through these APIs in near real-time fashion. SpectX retrieves audit events and reports depending on Microsoft’s data availability lag times and update frequency (see Advanced Configuration for details).

Note

When retrieved with SpectX, the log files are stored in the local file system [2]. It is the system administrator’s task to maintain the availability of disk space (archive older log files, etc).

Using msapi:// in SpectX

The msapi:// protocol does not allow anonymous access. To query the logs with SpectX, configure access by creating a new datastore and use the URI to that datastore in the queries.

SpectX uses OAuth 2.0 protocol for authentication and authorization with Microsoft API endpoints. SpectX needs to be registered in the Azure Management Portal and granted permissions to access the respective API’s and log data.

Hint

See step-by-step instructions for configuring access to Microsoft APIs.

Datastore configuration

UI

Configuration parameters for msapi:// datastore definition:

Name Description
Store name
Unique name among all defined DataStores
Application (client) ID
Application (client) ID assigned by Azure AD during
Directory (tenant) ID
ID of the tenant in the Azure AD, assigned during
Secret Key
Enabled Azure AD Logs
Enables/disables retrieving respective Azure AD audit log
Enabled Office 365 Logs
Enables/disables retrieving respective Office 365 audit log
Enabled Azure Activity Logs
Enables/disables retrieving respective Azure Activity audit log
ACL
specifies blob ACL

Microsoft makes Office 365 audit logs available based on subscription. Hence, for each enabled audit log type the subscription must be started (i.e status is enabled).

Advanced Configuration

Azure AD logs Settings

Name Description
Time limits
Specifies availability lag times and update frequencies [1]
for enabled Azure AD audit logs.
Request Rate Limit
Max request rate to apply for throttling, per query. Microsoft Graph API
enforces throttling per tenant (25 requests per 10 seconds), so should
you want to apply your own limits, use this parameter for that

Office 365 Settings

Name Description
Office 365 Plan
Type of Microsoft 365 or Office 365 subscription plan for your organization
Time limits
Specifies availability lag times and update frequencies [1]
for enabled Office 365 audit logs.
Request Rate Limit
Max request rate to apply for throttling, per query. Microsoft Office 365
Management Activity API applies throttling per tenant (60K requests/minute),
so should you want to apply your own limits, use this parameter for that

Azure Activity Logs Settings

Name Description
Lag Time
Specifies lag time [1]
Update Frequency
Specifies update frequenciey [1]
Request Rate Limit
Max request rate to apply for throttling, per query

Common settings

Name Description
Time Zone
The timezone to use for constructing audit log file URIs.
Default is the User Properties Timezone.
Hour in
Specifies whether locally stored log file timestamp hour value should be
included in the file name, path or both [2].
Log Type in
Specifies whether application name should be included in locally stored
file name, path, or both [2].
Connect Timeout
connect timeout. A timeout of zero is interpreted as an infinite timeout. Default
is 10s. A time interval evaluating to integer amount of milliseconds.
Read Timeout
read timeout. A timeout of zero is interpreted as an infinite timeout. Default is
60s. A time interval evaluating to integer amount of milliseconds.
Max Error Retries
a max number of times the SpectX tries to get access to a requested resource
in case it is inaccessible due to network problems until giving up.
The default is 3. A non-negative integer
[1](1, 2, 3, 4) The update frequency reflects how often an audit log is updated. The value is empirically determined and should not exceed the availability lag time. It is used to re-fetch an audit log which last fetch time is within the availability lag and the time since last fetch is bigger than update frequency.
[2](1, 2, 3) Locally stored log files are placed in the following directory structure: $SPECTX_HOME/engine_data/microsoft_api/<datastore_name>/<log_id>/<log_name>/yyyy/MM/dd/<HH>/yyyyMMdd<HH>.<log_name>.json. Here the <log_id> is an API identifier (azure_ad or office365), <log_name> is short name of the corresponding audit log provided by the API, and <HH> (hour value of the timestamp) may appear in the path or filename or both.

Filesystem

msapi:// datastore definition file is a JSON file with the following key-value pairs (optional parameters can be omitted):

{
  "type": "MSAPI",
  "msapiStore": {
    "clientId": "<ClientID>",
    "tenantId": "<TenantID>",
    "secret": "<SecretKey>",
    "timeZone" : "<timeZone>",
    "azureADTimeWindow" : "<azureADTimeWindow>",
    "azureTimeWindow" : "<azureTimeWindow>",
    "office365TimeWindow" : "<office365TimeWindow>",
    "fileNameIncludesApp" : <true|false>,
    "filePathIncludesApp" : <true|false>,
    "fileNameIncludesHour" : <true|false>,
    "filePathIncludesHour" : <true|false>,
    "azureADLogs" : "<azureADLogs>",
    "azureLogs" : "<azureLogs>",
    "office365Logs" : "<office365Logs>",
    "office365Plan" : "<office365Plan>",
    "azureADRateLimit" : <azureADRateLimit>,
    "azureADRateTimeunit" : "<azureADRateTimeunit>",
    "azureRateLimit" : <azureRateLimit>,
    "azureRateTimeunit" : "<azureRateTimeunit>",
    "office365RateLimit" : <office365RateLimit>,
    "office365RateTimeunit" : "<office365RateTimeunit>",
    "connectTimeout": <connectTimeout>,
    "readTimeout": <readTimeout>,
    "maxErrorRetries": <maxErrorRetries>,
    "userAgent": "<userAgent>",
    "acl": {<ACL>}
  }
}

where

  • <clientId> Application (client) ID assigned by Azure AD during application registration. A string. Mandatory parameter.
  • <tenantId> ID of the tenant in the Azure AD. A string. Mandatory parameter.
  • <secret> Client secret. A string. Mandatory parameter
  • <timeZone> time zone to use for constructing audit log file URIs. A string. Optional. Defaults to User Properties Timezone.
  • <azureADTimeWindow> a period from now specifying the time range of Azure AD audit events to retrieve. A time interval. Optional. Defaults to “30 days”.
  • <azureTimeWindow> a period from now specifying the time range of Azure Activity audit events to retrieve. A time interval. Optional. Defaults to “90 days”.
  • <office365TimeWindow> a period from now specifying the time range of Office 365 audit events to retrieve. A time interval. Optional. Defaults to “7 days”.
  • <fileNameIncludesApp> indicates if the log file name should include the audit log type when the URI is constructed. Boolean. Optional. Defaults to true.
  • <filePathIncludesApp> indicates if the log file path should include log type when the URI is constructed. Boolean. Optional. Defaults to false.
  • <fileNameIncludesHour> indicates if the log file name should include the timestamp’s hour value when the URI is constructed. Boolean. Optional. Defaults to false.
  • <filePathIncludesHour> indicates if the log file path should include the hour value when the URI is constructed. Boolean. Optional. Defaults to false.
  • <azureADLogs> Comma-separated list of colon-separated key-value pairs. Key specifies audit log “uri” name, value is composed of |-separated triple: Azure AD Activity reports audit log type (audits or signins), expected lag time (time interval) and up-to-date-age (time interval). A string. Optional.
  • <azureLogs> Comma-separated list of colon-separated key-value pairs. Key specifies audit log “uri” name, value is composed of |-separated triple: Azure Activity audit log type (audits, signins, ni-signins, sp-signins, mi-signins), expected lag time (time interval) and up-to-date-age (time interval). A string. Optional.
  • <office365Logs> Comma-separated list of colon-separated key-value pairs. Key specifies audit log “uri” name, value is composed of |-separated triple: Office 365 audit log type (Audit.AzureActiveDirectory, Audit.Exchange, Audit.SharePoint, Audit.General, DLP.All), expected lag time (time interval) and up-to-date-age (time interval). A string. Optional.
  • <office365Plan> Type of Microsoft 365 or Office 365 subscription plan for your organization (ent for Enterprise plan, gcc for GCC government plan, gcc_high for GCC High government plan, dod for DoD government plan). Optional, default is ent
  • azureADRateLimit and azureADRateTimeunit specify correspondingly the max rate and time interval of the request throttling to apply to Azure AD requests (i.e. azureADRateLimit: 25, azureADRateTimeunit: "10 sec". Optional
  • azureRateLimit and azureARateTimeunit specify correspondingly the max rate and time interval of the request throttling to apply to Azure Activity logs requests (i.e. azureRateLimit: 25, azureRateTimeunit: "10 sec". Optional
  • office365RateLimit and office365RateTimeunit specify correspondingly the max rate and time interval of the request throttling to apply to Azure AD requests (i.e. office365RateLimit: 1000, office365RateTimeunit: "1 sec". Optional
  • <connectTimeout> Connection timeout. A timeout of zero is interpreted as an infinite timeout. Optional. The default is 10s. A time interval evaluating to integer amount of milliseconds.
  • <readTimeout> Read timeout. A timeout of zero is interpreted as an infinite timeout. Optional. The default is 60s. A time interval evaluating to integer amount of milliseconds.
  • <maxErrorRetries> A max number of times SpectX tries to get access to a requested resource before giving up in case it is inaccessible due to network problems. Optional. A non-negative integer. The default is 3.
  • <userAgent> is a value for software agent name to be used when communicating with the Microsoft APIs. Optional. A string. Default value is composed of a string “SpectX” and a current software version designator.
  • <ACL> is a definition of a blob ACL for the datastore. A map.