CLOUD

Splunking Microsoft Azure Monitor Data – Part 1 – Azure Setup

Azure Monitor is a platform service that provides data routing and access for Azure resources.  Azure Monitor exposes 3 main types of data:

1) Metrics – these are typically performance metrics

2) Diagnostic Logs – logs generated by a resource

3) Activity Logs – who did what and when in the Azure environment

In order to get this data into Splunk, certain setup steps need to happen on both the Azure side and the Splunk side.  This blog post focuses on what needs to happen on the Azure side.  We will look at the Azure Monitor Add-on for Splunk for data collection in a later blog post.

Pro tip: A PowerShell script is available that can automate the Azure setup.

How Azure Monitor Data is Exposed

Activity logs are generated by the Azure control-plane.  This is basically who did what and when in your Azure environment as well as information like service health, recommendations, security and more.  Azure Monitor routes activity log data to an Event Hub

Diagnostic logs are generated by a resource after it is provisioned.  The contents of a diagnostic log will differ depending on what type of resource was provisioned.  As an example, a Key Vault diagnostic log will contain an event when a secret is retrieved via a "Get" operation.  Diagnostic data can be routed to more than one place, but for the purposes of this blog, the data will be routed to an Event Hub.

Metrics are numerical values generated by an Azure resource to help you understand the operation and performance of the resource.  Metrics are accessible via a REST API and are available for 93 days (as of this writing). 

Azure Components

There are three main components needed on the Azure side:

1) Azure AD Application – I like to think of this as a user or service account with a user ID and password - except an Azure AD application has an application ID and key.

2) Event Hub – this is the central location where activity and diagnostics logs will be stored.

3) Key Vault – the application and Event Hub Namespace keys will be stored here (so, a Key Vault is pretty much what it sounds like).

We will be granting the Azure AD application certain roles and permissions during the setup process.  This is done by creating service principal objects.

Going back to the user analogy, after creating a user, you often need to grant them permissions to areas in your network.  To do this for an Azure AD application, you create a service principal object.  The service principal defines the access policy and permissions for the application in a tenant.  So, an Azure AD application is like a user (with an ID and password), and a service principal object is like the user's level of access.  This is an oversimplification of course, so here are the technical details.

Event Hub Security

In order to programmatically pull data from an Event Hub into Splunk, you need a shared access policy name and key.  These values are not stored in the Azure Monitor Add-on for Splunk.  Instead, these values will be retrieved from a Key Vault.

 

Metrics Security

Metric data is pulled from Azure via a REST API.  An Azure AD application ID and key are needed to access the Azure Metric data.  Similar to the Event Hub, these values are not stored in the Azure Monitor Add-on for Splunk.  Instead, these values will be retrieved from a Key Vault.

 

Key Vault Security

The Azure Monitor Add-on for Splunk will not store the Azure AD application key or the Event Hub Namespace key.  Instead, these values will be stored in a Key Vault.  However, in order to get the keys mentioned above out of the Key Vault programmatically, you have to authenticate to Azure.  This is done using (you guessed it) an Azure AD application ID and key.  This Azure AD application ID and key WILL BE stored by the Azure Monitor Add-on for Splunk.  You can reuse the same Azure AD application used above, or create a new one with "Get" permissions in the Key Vault.  For the purpose of this blog, we will reuse the same Azure AD application for both.

 

Setup Overview

Now that you know what Azure components are needed, let’s look at an overview of the setup:

1) Create an Azure AD application registration.  Every time the Azure Monitor Add-on for Splunk asks for data from Azure, it will do so in the context of an Azure AD application.

2) Create an Event Hub Namespace.  Azure will write activity logs and diagnostics logs to individual Event Hubs contained with this namespace.  You are not limited to one namespace, but for the purposes of this blog, we will just use one.

3) Create a Key Vault.  A Key Vault will hold the keys for the Azure AD application and the Event Hub.

4) Set up all necessary Service Principal objects (think of these like the rights a user has on a network).  Since Splunk will be getting data from Azure in the context of the Azure AD application, we need to make sure that application has the necessary rights to the sections of Azure that house the data.

5) Store the Azure AD application and Event Hub Namespace keys in the Key Vault.

Along the way in this guide, some pieces of information will need to be saved for later – either for the Azure setup or the Splunk setup.  Those pieces of information will be called out in each step with this icon 

Ok, let's set everything up on the Azure side now…

Create an Azure AD Application

1. Select Azure Active Directory -> App registrations -> New application registration

Note: an existing Azure AD application registration may be used if desired.

2. Enter a name and Sign-on URL and click Create. 

Note: the Sign-on URL is not used, so any valid URL is acceptable.

3. Create a Key for the Application Registration

Click Settings -> Keys -> enter a key description -> set the expiration for the key -> click Save.

 

After clicking the save button, make a copy of the value for the key.  This is the only time the value will be displayed in the Azure portal.  If you lose this value, a new key will need to be generated using the above process.  This key will be stored in a Key Vault later.

 

 Save the Application ID and Key.

 Application ID  11111111-1111-1111-1111-111111111111    
 Application Key  22222222-2222-2222-2222-222222222222    

Grant the Azure AD Application Reader Access to the Subscription

After creating the Azure AD application (which is similar to a service account), the application needs to be granted reader access to the subscription(s) via a service principal object.

1. Select Subscriptions -> your subscription -> Access control (IAM) -> Add -> select the Reader role -> type the name of your application registration from the previous step -> select the application when it appears in the results -> click the Save button.

If you have multiple subscriptions that you would like to use with Splunk, repeat the above step for each subscription.

Create an Event Hub Namespace

The Event Hub Namespace will contain one or more Event Hubs.  The configured Azure services will create Event Hubs in this namespace to store activity logs and diagnostics logs.

1. Select Event Hubs -> Add -> supply a Name, Resource group, and any other settings -> Create

2. After the Event Hub Namespace is created, we will need to get the key to store in the Key Vault

Select your Event Hub Namespace -> Shared access policies -> RootManageSharedAccessKey -> copy the Primary key

 Save the policy name and primary key.

 Application ID  11111111-1111-1111-1111-111111111111  
 Application Key  22222222-2222-2222-2222-222222222222 
 Event Hub Namespace      example: splunkdev
 Event Hub Policy Name      RootManageSharedAccessKey
 Event Hub Primary Key      1234asdf4321fdsa1234asdf4321fdsa

Create a Key Vault

The Azure Monitor Add-on for Splunk will not store the Azure AD application key (similar to a password) or Event Hub Namespace key.  Instead, it wil retrieve this information from a Key Vault.

1. Key vaults -> Add -> enter a Name, Resource Group, and any other settings -> Create

Microsoft Azure Add Event Hub

Note: By default, the current user will be granted an access policy that can perform most actions in the Key Vault.  We will add an additional access policy in the next step for our Azure AD application.

2. Select Key vaults -> your key vault -> Access policies -> Add new

3. Click Select principal -> type the name of the Azure AD application registration -> click the name of the Azure AD application registration in the results -> Select.

4. Select "Get" from the Secret permissions list -> OK.

5. Save

 Save the Key Vault Name

 Application ID  11111111-1111-1111-1111-111111111111  
 Application Key  22222222-2222-2222-2222-222222222222 
 Event Hub Namespace      example: splunkdev
 Event Hub Policy Name      RootManageSharedAccessKey
 Event Hub Primary Key      1234asdf4321fdsa1234asdf4321fdsa
 Key Vault Name  Your Key Vault name

Store the Event Hub Namespace and Azure AD Application Keys in the Key Vault

1. Select Key vaults -> your key vault -> Secrets -> Generate/Import

2. Enter a name -> paste the value of the Event Hub Namespace primary key in the value -> enter RootManageSharedAccessKey in the content type -> Create.

Create a Secret Event Hub Key

3. Click the created secret -> current version -> copy the secret version

 

4. Repeat this step for the Azure AD application's key.

Enter a name -> paste the key in to the value -> paste the application ID into the content type -> Create.

Create a Secret Application Key

5. Click the created secret -> current version -> copy the secret version

 Application ID  11111111-1111-1111-1111-111111111111  
 Application Key  22222222-2222-2222-2222-222222222222 
 Event Hub Namespace      example: splunkdev
 Event Hub Policy Name      RootManageSharedAccessKey
 Event Hub Primary Key      1234asdf4321fdsa1234asdf4321fdsa
 Key Vault Name  Your Key Vault name
 Event Hub Key Secret Name  example: myEventHubKey
 Event Hub Key Secret Version  1234asdf4321fdsa1234asdf4321fdsa
 Application Key Secret Name  example: myAppKey
 Application Key Secret Version  1234asdf4321fdsa1234asdf4321fdsa

Recap

We now have all the plumbing set up, so let's take a moment to recap what has been done so far:

1) Created an Azure AD application registration.  This is similar to a user or service account where the Application ID is like a user ID and the application key is like a password.

2) Granted the Azure AD application reader access to our subscription(s) via a Service Principal object.

3) Created an Event Hub Namespace.

4) Created a Key Vault.

5) Granted the Azure AD application Get access to the secrets stored in the Key Vault.

6) Stored the Azure AD application and Event Hub Namespace keys in the Key Vault.

Now, we will expose some data in Azure

Send Activity Logs to an Event Hub

1. Monitor -> Activity log -> Export

2. Select your subscription and regions to export -> set a retention -> check the "Export to an event hub" box -> Service bus namespace -> select the Event Hub Namespace and policy name created earlier -> OK -> Save

Send Diagnostic Logs to an Event Hub

Diagnostic logs are configured at the resource level and can be configured via the Azure Portal, PowerShell, or Azure CLI.  Here are more details on enabling diagnostic log settings.

As an example, we will enable diagnostics logs for the Event Hub we created earlier.

1. Event Hubs -> select your Event Hub -> Diagnostics logs -> Turn on diagnostics

2. Enter a name -> check the Stream to an event hub box -> select your Event Hub and access policy -> OK -> select which logs you would like to send to the Event Hub -> Save

Prepare for Metric Data Collection

With activity logs and diagnostic logs, it has been an "all or nothing" situation – meaning you turn it on or off.  Metrics is a different story though as you have the ability to select which metrics you want to collect.  To configure which metric you want to collect, create a tag named "Metrics" on the resource.  Here is an example using our Event Hub created earlier.

1. Tags -> enter "Metrics" for the tag name -> enter an asterisk * to collect all available metrics for the resource, or coma-separated list of specific metric names for the value -> Save

Here is a list of available metrics by resource type is available.

Note 1: When specifying metric names explicitly, use the "Metric Display Name".  For example, use Incoming Requests instead of IncomingRequests

Note 2: "It may seem expedient to simply set Metrics:* for all resources and let the add-on sort out the details. And in a perfect world, this would work fine. The add-on doesn't care. However, there are limits and throttles around just about everything in computing. One of these is the length of the query string that can be sent to the ARM api's. The Redis Cache resource has quite a number of available metrics (for example). Requesting all available metrics for Redis Cache blows past the query string limits (2048 bytes). Best practice is in cases like this to limit the number of metrics requested to a top-level set for the typical case, then drill down as the situation demands."

Source -> https://github.com/Microsoft/AzureMonitorAddonForSplunk/wiki/Configuration-of-Azure

Conclusion

This wraps up everything that needs to happen on the Azure side to make data readily ingestible to Splunk via the Azure Monitor for Splunk Add-on.  We have set up an Azure AD application, Event Hub Namespace, Key Vault, Service Principals, Activity log settings, Diagnostics log settings, and Metric collection tags.  In the next blog post, we will look at installing the Azure Monitor Add-on for Splunk and setting up data inputs to pull down all of the data we made available in Azure.

Jason Conger
Posted by

Jason Conger

TAGS

Splunking Microsoft Azure Monitor Data – Part 1 – Azure Setup

Show All Tags
Show Less Tags

Join the Discussion