Managing SNMP Traps with ITSI Event Analytics

Splunk IT Service Intelligence (ITSI) lets you gain unique business and service-relevant context using your machine data. But while ITSI continues to push event analytics and service insights into the next generation, it can also breathe new life into traditional data sources, making them more accessible and valuable for everyone.

Enter SNMP.

To most, SNMP (Simple Network Management Protocol) is a foul four-letter word; but those who earned their stripes in network operations know that it remains a staple of observing a network.

SNMP is a well-defined protocol that contains tons of useful context and information from vendors that can be used to enrich data in Splunk and ITSI. When used to its full potential, it can drive key insights into your environment.

Traditionally, SNMP data has been difficult to obtain and understand, thus hindering its value outside your Network Operations teams. However, Splunk’s goal has always been to make data accessible and valuable to all. This blog post focuses on receiving SNMP traps and feeding them into ITSI Notable Events Review for easy review and correlation.

In this configuration, ITSI makes use of snmptrapd as a data source. Snmptrapd is an application that receives and logs SNMP “trap” and “inform” messages, resolves them using vendor MIBs (Management Information Bases), then writes to disk where our universal forwarder is ready and waiting. This pattern is very similar to the well-known Syslog approaches that Splunkers have been using for years.

Install and Configure snmptrapd

Although the Splunk documentation points to the use of snmptrapd as a method of receiving traps, this post digs deeper. It shows how simple configurations can turn cryptic SNMP messages into notable events that are easy to read and search in ITSI.

1. Install snmptrapd on CentOS 7

This demonstration uses an AWS Amazon Machine Image (AMI) for CentOS 7:

Once the AMI is ready, I have installed some basic tools like Wget and cURL, and disabled SELinux for ease of demonstration. Proper configuration of SELinux is beyond the scope of this blog post, but it's recommended that you review your security requirements once you implement the solution.

Let’s get started!

Run the following command to install the SNMP packages:

yum install net-snmp-utils net-snmp-libs net-snmp

To review what packages you have, run the following command:

rpm -qa| grep snmp

The output should look something like this:

[root@ip-172-31-31-57 log]# rpm -qa| grep snmp

2. Configure snmpd

snmpd is an SNMP agent that binds to a port and awaits requests from SNMP management software. Upon receiving a request, it processes the request, collects the requested information and/or performs the requested operation and returns the information to the sender. 

To start the localhost snmpd configuration wizard, run the following command:

snmpconf -r none -i -g basic_setup

This wizard is handy to generate your snmpd.conf file. Use the file to customize your snmptrapd deployment, which will be stored at /usr/share/snmp/snmpd.conf.

Note: If you prefer, you can edit snmpd.conf directly instead.

3. Run snmptrapd at boot

Set the snmptrapd agent to run at boot time by running the following command:

systemctl enable snmptrapd

4. Configure snmptrapd startup flags

Now let’s configure snmptrapd startup flags. These flags tell snnptrapd where the configuration lives, where to write the log, what MIBS to load, etc. For more information about the available options, see the snmptrapd documentation.

Run the following command to open the configuration file:

vi /etc/sysconfig/snmptrapd

Add the following lines, then save and close the file:

# snmptrapd command line options
# '-f' is implicitly added by snmptrapd systemd unit file
# OPTIONS="-Lsd"
OPTIONS="-c /etc/snmp/snmptrapd.conf -A -n -Lf /var/log/snmptrapd.log -OQ -m 
ALL --disableAuthorization=yes -p /var/run/"

5. Configure snmptrapd logging format

To make your snmptrapd.log easy to work with and parse, use an easy-to-read format. The key is to use key-value pairs that Splunk can easily handle without needing excessive custom parsing rules.

Open snmptrapd.conf:

vi /etc/snmp/snmptrapd.conf 

Add the following configuration to format traps for easy parsing, then save and close the file:

# snmptrapd formatting
format1 Agent_Address = %A\nAgent_Hostname = %B\nDate = %y-%02.2m-%02.2l 
%02.2h:%02.2j:%02.2k\nEnterprise_OID = %N \nTrap_Type = %w\nTrap_SubType = 
%q\nCommunity_Infosec_Context = %P\nUptime = %T\nDescription = 
format2 Agent_Address = %A\nAgent_Hostname = %B\nDate = %y-%02.2m-%02.2l 
%02.2h:%02.2j:%02.2k\nEnterprise_OID = %N\nTrap_Type = %w\nTrap_SubType = 
%q\nCommunity_Infosec_Context = %P\nUptime = %T\nDescription = 

Now change directories to /var/log and create an snmptrapd.log file.

To change directories, run the following command:

cd /var/log

Create the snmptrapd.log file:

touch snmptrapd.log


6. Configure snmptrapd to parse MIBS from 3rd party vendors

The ability of snmptrapd to translate traps using SNMP MIBs is essential to this solution. Ensure that you load up all MIBS that you expect to receive into a directory on the host. For more information, see Configuring snmptrapd to parse MIBS from 3rd party Vendors.

To find out which directories are used on your system, run the following command:

net-snmp-config --default-mibdirs

The output should look like this:

net-snmp-config --default-mibdirs

This output confirms the location where we will be loading custom MIBs, in this case /usr/share/snmp/mibs. At this point, copy over any MIBs you have from your vendors into this directory. For more information, see Using and loading MIBS.

In this example, we have loaded a few MIBs from Juniper, Cisco, and Eaton UPS. Speak with your engineers and architects to ensure that you get the whole library of MIBs your vendors have provided over the years.

To restart the snmptrapd daemon, run the following command:

systemctl restart snmptrapd

Check the status:

systemctl status -l snmptrapd

The output should look something like this. Take note of the startup flags:

[root@ip-172-31-31-57 snmp]# systemctl status -l snmptrapd
● snmptrapd.service - Simple Network Management Protocol (SNMP) Trap Daemon.
  Loaded: loaded (/usr/lib/systemd/system/snmptrapd.service; disabled; vendor preset: disabled)
  Active: active (running) since Tue 2018-08-07 22:15:04 UTC; 5min ago
Main PID: 1679 (snmptrapd)
  CGroup: /system.slice/snmptrapd.service
          └─1679 /usr/sbin/snmptrapd -c /etc/snmp/snmptrapd.conf -A -n -Lf 
/var/log/snmptrapd.log -OQ -m ALL --disableAuthorization=yes -p 
/var/run/ -f
Aug 07 22:15:04 systemd[1]: 
Starting Simple Network Management Protocol (SNMP) Trap Daemon....
Aug 07 22:15:04 systemd[1]: 
Started Simple Network Management Protocol (SNMP) Trap Daemon..
[root@ip-172-31-31-57 snmp]#

Send a Test Trap

Now that we have our snmptrapd daemon running, let’s test it!

To send a test trap, run the following command:

snmptrap -v 2c -c public <destination_hostname> '' i 123456

If you completed the installation and configuration step correctly, you should now see your trap in /var/log/snmptrapd.log (or wherever you have chosen to store the logs).

Tail -100 snmptrapd.log
Agent_Address =
Agent_Hostname = UDP: []:23735->[]:162
Date = 2018-08-17 16:23:10
Enterprise_OID = .
Trap_Type = 0
Trap_SubType = 0
Community_Infosec_Context = TRAP2, SNMP v2c, community c4ct1
Uptime = 0
Description = Cold Start
DISMAN-EVENT-MIB::sysUpTimeInstance = 37:20:05:56.58
SNMPv2-MIB::snmpTrapOID.0 = NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification
NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate = 123456

As you can see, our traps are translated and logged in a readable fashion. Splunk should have no problem handling them as key-value pairs.

Configure Splunk

Use the following configurations to instruct Splunk to monitor the file that snmptrapd is writing, and to parse and extract the key-value pairs that make up key information from the traps you have received.

It is recommended that you bundle these configuration files inside ITSI. Store them in $SPLUNK_HOME/etc/apps/SA-ITOA/local.

Note: Where you put the following configurations depends on your Splunk architecture. For more information, see About installing Splunk add-ons in the Splunk Add-ons Manual.

1. Configure inputs.conf

Add the following stanza to the local version of inputs.conf:

disabled = false
index = snmptrapd
sourcetype = snmptrapd

This stanza tells Splunk where to look for our data. You only need to deploy this configuration on forwarders or Splunk instances co-located with snmptrapd.

2. Configure props.conf

Add the following stanza to the local version of props.conf:

KV_MODE = none
LINE_BREAKER = ([\r\n]+)Agent_Address\s=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = Date\s=\s
category = Custom
description = parse snmptrapd logging with custom kvpair splunk formatting
disabled = false
pulldown_type = true
EXTRACT-node = ^[^\[\n]*\[(?P<node>[^\]]+)
REPORT-snmptrapd = snmptrapd_kv

This stanza sets line breaking rules and extracts the timestamp. In this case, the timestamp is the time the snmptrapd server received the trap. It also ensures that Splunk knows how to parse the time format.

We perform field extractions, one being a ‘node’ field, the other pointing to a transform that will extract all key-value pairs in the trap.

Deploy this configuration to your heavy forwarder, indexer, and/or search head.

3. Configure transforms.conf

Add the following stanza to the local version of transforms.conf:

DELIMS = "\n,"  = "

Deploy this configuration to your heavy forwarder, indexer, and/or search head.

4. Configure local.meta.conf

Add the following stanza to the local version of default.meta.conf:

access = read : [ * ], write : [ admin ]
export = system

This stanza includes a local.meta.conf file in the ITSI application, which exports permissions to the system. This ensures that ITSI can use our extractions. This setting is very important for ITSI correlation searches.

Deploy this configuration on your search heads.

5. Test the configuration in Splunk

If deployed successfully, you will see your traps in Splunk. Splunk should be extracting a ‘node’ field, and all key-value pairs in the traps:

Configure an ITSI Correlation Search

After you set up snmptrapd, configure how SNMP traps show up in ITSI Notable Events Review. Add the following correlation search in ITSI:

This simple correlation search brings in the raw snmptrapd data and presents it in Notable Events Review. It uses the ‘node’ and ‘Description’ fields for Titles/Descriptions and provides simple drilldown logic to the node.

Configure an ITSI Aggregation Policy

Use the same fields you used in the correlation search to configure the following aggregation policy in ITSI. The policy aggregates SNMP traps so they are displayed in a grouped format in Notable Events Review:

Test the Configuration

Return to ITSI Notable Events Review for a great initial look at the noise reduction you’ll see on your incoming SNMP data. The aggregated notable event group provides context and actionable data that your teams can easily manage.

From thousands of traps to a handful of actionable events!

After completing the steps outlined above, you can successfully use snmptrapd to receive SNMP traps and feed them into ITSI Notable Events Review. With the power of ITSI Event Analytics, you can use Splunk ITSI to its full potential to make your SNMP traps accessible, readable, and valuable to everyone. 

Liz Snyder

Posted by