CLOUD

Splunking Microsoft Azure Monitor Data – Part 2 – Splunk Setup

Azure Monitor is a platform service that provides data routing and access for Azure resources. Azure Monitor exposes 3 main types of data:

  1. Metrics – these are typically performance metrics
  2. Diagnostic Logs – logs generated by a resource
  3. Activity Logs – who did what and when in the Azure environment

In order to get this data into Splunk, certain setup steps need to happen on both the Azure side and the Splunk side. My previous blog post, "Splunking Microsoft Azure Monitor Data – Part 1 – Azure Setup," covered the steps necessary on the Azure side for Azure Monitor data collection. This blog post focuses on what needs to happen on the Splunk side.

Setup Overview

Fortunately, there is an Azure Monitor Add-on for Splunk that adds inputs for all three types of data mentioned above. However, there are a couple extra steps involved in setting up this add-on. The following is an overview of what we will be detailing in this blog post:

  1. Download and install Azure Monitor Add-on for Splunk
  2. How the Azure Monitor Add-on for Splunk get data
  3. Set up an input for Activity Logs
  4. Set up an input for Diagnostics Logs
  5. Set up an input for Metrics

Download and Install the Azure Monitor Add-on for Splunk

The Azure Monitor Add-on for Splunk is listed on Splunkbase, but hosted externally as open source on Microsoft's public GitHub organization. The setup instructions are available in the README.md file as well as the Wiki and are very detailed, so I won’t rehash them here.

Note: Be sure to follow the directions for downloading the Python and Node.js dependencies.

How the Azure Monitor Add-on for Splunk Gets Data

If you were following along in the previous blog post, you would have ended up with a table of saved information like the following:

 Appliction ID  11111111-1111-1111-1111-111111111111
 Application Key  22222222-2222-2222-2222-222222222222
 Event Hub Namespace  example: splunkdev
 Event Hub Policy Name  RootManageSharedAccessKey
 Event Hub Primary Key  1234asdf4321fdsa1234asdf4321fdsa
 Key Vault Name  Your Key Vault name
 Event Hub Key Secret Name  example: myEventHubKey
 Event Hub Key Secret Version  1234asdf4321fdsa1234asdf4321fdsa
 Application Key Secret Name  example: myAppKey
 Appliction Key Secet Version  1234asdf4321fdsa1234asdf4321fdsa

Recall that activity logs and diagnostics logs are going to be routed to an Event Hub by Azure Monitor. In order to access the Event Hub data programmatically, you need an 1) Event Hub Namespace, 2) policy name, and 3) policy key. The Event Hub policy name and policy key are stored in a Key Vault in Azure instead of with the add-on. The Azure Monitor Add-on for Splunk will retrieve the policy name and policy key from the Key Vault on each run.

This same methodology is used for REST API access to get Metric data. In order to access the Metric data programmatically via the REST API, you need 1) an Azure AD application ID and 2) application key. The Azure AD application ID and key are stored in a Key Vault in Azure instead of with the add-on. The Azure Monitor Add-on for Splunk will retrieve the Azure AD application ID and key from the Key Vault on each run.

In order to get the secrets from the Key Vault, the add-on has to authenticate to Azure. This is done using a separate Azure AD application ID and key that has "Get" access to the secrets in the Key Vault. The Azure AD application ID and key used here can be the same as the Azure AD application ID and key used to access the Metrics REST API. But, you may use two separate Azure AD application IDs and keys specific for each task.

Set Up an Input for Activity Logs

After installing the add-on in your Splunk environment, go to Settings -> Data inputs

Click the Azure Monitor Activity Log input -> New

The following screen is presented prompting for input parameters:

We have most of this data already in our table from the previous blog post:

Item

 Add-on parameter name

Application ID

 SPNApplicationId

Application Key

 SPNApplicationKey

Event Hub Namespace

 eventHubNamespace

Key Vault Name

 vaultName

Event Hub Key Secret Name

 secretName

Event Hub Key Secret Version

 secretVersion


There is one more piece of information we are missing, and that is the SPNTenantID. This is your Azure AD Directory ID.

1. From the Azure Portal: Azure Active Directory -> Properties -> Directory ID

After entering all the required parameters, click the Next button.

Click the Start Searching button.

The following search will enumerate the types of Azure Monitor activity logs ingested:

sourcetype=amal* | stats values(sourcetype)

Notice that the Activity Log input creates different sourcetypes based on the data ingested.

Here is a search that shows the creation of the Event Hub namespace from the previous blog post:

sourcetype="amal:administrative" operationName="MICROSOFT.EVENTHUB/NAMESPACES/WRITE"

Set Up an Input for Diagnostics Logs

The process for setting up a diagnostics log input is similar to setting up an activity log input as outlined above:

  1. Settings -> Data inputs
  2. Azure Monitor Diagnostic Logs -> New
  3. Supply the input parameters -> Next
  4. Start Searching

In the previous blog, we turned on Event Hub diagnostics logs. Here is a search to display the data:

source="azure_diagnostic_logs:*"

Here is a result from the above search showing the "Get" operation on a Key Vault secret:

Set Up an Input for Metrics

The process for setting up a Metrics input is similar to setting up the other inputs as outlined above:

  1. Settings -> Data inputs
  2. Azure Monitor Metrics -> New
  3. Supply the input parameters -> Next
  4. Start Searching

We have most of this data already in our table from the previous blog post:

Item

 Add-on parameter name

Application ID

 SPNApplicationId

Application Key

 SPNApplicationKey

Event Hub Namespace

 eventHubNamespace

Key Vault Name

 vaultName

Application Key Secret Name

 secretName

Application Key Secret Version

 secretVersion

 

There is one more piece of information we are missing, and that is the SubscripitonId. To get this from the Azure Portal:

1. Subscriptions -> subscription name -> copy the Subscription Id

In the previous blog, we tagged our Event Hub to collect Metrics. Here is a search to display the data:

sourcetype="amm:eventhub:namespace"

Here is a result from the above search showing Incoming Requests:

Notice how the add-on automatically set the sourcetype to amm:eventhub:namespace. This is controlled by a file named sourcetypes.json located in the $SPLUNK_HOME/etc/apps/TA-Azure_Monitor/bin directory. You can modify this file as needed.

Conclusion

In this post, we looked at how to install the Azure Monitor Add-on for Splunk, how to configure the 3 inputs (Activity Logs, Diagnostics Logs, and Metrics), and provided a few details as to the add-on’s operation. Additional detail about configuring this add-on can be found on the public repository Wiki. In the next post, we will look at use cases for the data gathered by the Azure Monitor Add-on for Splunk.

Jason Conger
Posted by

Jason Conger

Join the Discussion