Splunk IT Service Intelligence (ITSI) lets you gain unique business and service-relevant context using your machine data. But while ITSI continues to push event analytics and service insights into the next generation, it can also breathe new life into traditional data sources, making them more accessible and valuable for everyone.
To most, SNMP (Simple Network Management Protocol) is a foul four-letter word; but those who earned their stripes in network operations know that it remains a staple of observing a network.
SNMP is a well-defined protocol that contains tons of useful context and information from vendors that can be used to enrich data in Splunk and ITSI. When used to its full potential, it can drive key insights into your environment.
Traditionally, SNMP data has been difficult to obtain and understand, thus hindering its value outside your Network Operations teams. However, Splunk’s goal has always been to make data accessible and valuable to all. This blog post focuses on receiving SNMP traps and feeding them into ITSI Notable Events Review for easy review and correlation.
In this configuration, ITSI makes use of snmptrapd as a data source. Snmptrapd is an application that receives and logs SNMP “trap” and “inform” messages, resolves them using vendor MIBs (Management Information Bases), then writes to disk where our universal forwarder is ready and waiting. This pattern is very similar to the well-known Syslog approaches that Splunkers have been using for years.
Install and Configure snmptrapd
Although the Splunk documentation points to the use of snmptrapd as a method of receiving traps, this post digs deeper. It shows how simple configurations can turn cryptic SNMP messages into notable events that are easy to read and search in ITSI.
1. Install snmptrapd on CentOS 7
This demonstration uses an AWS Amazon Machine Image (AMI) for CentOS 7:
Once the AMI is ready, I have installed some basic tools like Wget and cURL, and disabled SELinux for ease of demonstration. Proper configuration of SELinux is beyond the scope of this blog post, but it's recommended that you review your security requirements once you implement the solution.
Let’s get started!
Run the following command to install the SNMP packages:
yum install net-snmp-utils net-snmp-libs net-snmp
To review what packages you have, run the following command:
rpm -qa| grep snmp
The output should look something like this:
[root@ip-172-31-31-57 log]# rpm -qa| grep snmp net-snmp-5.7.2-33.el7_5.2.x86_64 net-snmp-libs-5.7.2-33.el7_5.2.x86_64 net-snmp-agent-libs-5.7.2-33.el7_5.2.x86_64 Net-snmp-utils-5.7.2-33.el7_5.2.x86_64
2. Configure snmpd
snmpd is an SNMP agent that binds to a port and awaits requests from SNMP management software. Upon receiving a request, it processes the request, collects the requested information and/or performs the requested operation and returns the information to the sender.
To start the localhost snmpd configuration wizard, run the following command:
snmpconf -r none -i -g basic_setup
This wizard is handy to generate your snmpd.conf file. Use the file to customize your snmptrapd deployment, which will be stored at /usr/share/snmp/snmpd.conf.
Note: If you prefer, you can edit snmpd.conf directly instead.
3. Run snmptrapd at boot
Set the snmptrapd agent to run at boot time by running the following command:
systemctl enable snmptrapd
4. Configure snmptrapd startup flags
Now let’s configure snmptrapd startup flags. These flags tell snnptrapd where the configuration lives, where to write the log, what MIBS to load, etc. For more information about the available options, see the snmptrapd documentation.
Run the following command to open the configuration file:
Add the following lines, then save and close the file:
# snmptrapd command line options # '-f' is implicitly added by snmptrapd systemd unit file # OPTIONS="-Lsd" OPTIONS="-c /etc/snmp/snmptrapd.conf -A -n -Lf /var/log/snmptrapd.log -OQ -m ALL --disableAuthorization=yes -p /var/run/snmptrapd.pid"
5. Configure snmptrapd logging format
To make your snmptrapd.log easy to work with and parse, use an easy-to-read format. The key is to use key-value pairs that Splunk can easily handle without needing excessive custom parsing rules.
Add the following configuration to format traps for easy parsing, then save and close the file:
# snmptrapd formatting #http://www.net-snmp.org/wiki/index.php/TUT:Configuring_snmptrapd_to_parse_MIBS _from_3rd_party_Vendors # SNMPV1 format1 Agent_Address = %A\nAgent_Hostname = %B\nDate = %y-%02.2m-%02.2l %02.2h:%02.2j:%02.2k\nEnterprise_OID = %N \nTrap_Type = %w\nTrap_SubType = %q\nCommunity_Infosec_Context = %P\nUptime = %T\nDescription = %W\nPDU_Attribute_Value_Pair_Array:\n%V\n%v\n---\n # SNMPV2 format2 Agent_Address = %A\nAgent_Hostname = %B\nDate = %y-%02.2m-%02.2l %02.2h:%02.2j:%02.2k\nEnterprise_OID = %N\nTrap_Type = %w\nTrap_SubType = %q\nCommunity_Infosec_Context = %P\nUptime = %T\nDescription = %W\nPDU_Attribute_Value_Pair_Array:\n%V\n%v\n---\n
Now change directories to /var/log and create an snmptrapd.log file.
To change directories, run the following command:
Create the snmptrapd.log file:
6. Configure snmptrapd to parse MIBS from 3rd party vendors
The ability of snmptrapd to translate traps using SNMP MIBs is essential to this solution. Ensure that you load up all MIBS that you expect to receive into a directory on the host. For more information, see Configuring snmptrapd to parse MIBS from 3rd party Vendors.
To find out which directories are used on your system, run the following command:
The output should look like this:
net-snmp-config --default-mibdirs /root/.snmp/mibs:/usr/share/snmp/mibs
This output confirms the location where we will be loading custom MIBs, in this case /usr/share/snmp/mibs. At this point, copy over any MIBs you have from your vendors into this directory. For more information, see Using and loading MIBS.
In this example, we have loaded a few MIBs from Juniper, Cisco, and Eaton UPS. Speak with your engineers and architects to ensure that you get the whole library of MIBs your vendors have provided over the years.
To restart the snmptrapd daemon, run the following command:
systemctl restart snmptrapd
Check the status:
systemctl status -l snmptrapd
The output should look something like this. Take note of the startup flags:
[root@ip-172-31-31-57 snmp]# systemctl status -l snmptrapd ● snmptrapd.service - Simple Network Management Protocol (SNMP) Trap Daemon. Loaded: loaded (/usr/lib/systemd/system/snmptrapd.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2018-08-07 22:15:04 UTC; 5min ago Main PID: 1679 (snmptrapd) CGroup: /system.slice/snmptrapd.service └─1679 /usr/sbin/snmptrapd -c /etc/snmp/snmptrapd.conf -A -n -Lf /var/log/snmptrapd.log -OQ -m ALL --disableAuthorization=yes -p /var/run/snmptrapd.pid -f Aug 07 22:15:04 ip-172-31-31-57.ca-central-1.compute.internal systemd: Starting Simple Network Management Protocol (SNMP) Trap Daemon.... Aug 07 22:15:04 ip-172-31-31-57.ca-central-1.compute.internal systemd: Started Simple Network Management Protocol (SNMP) Trap Daemon.. [root@ip-172-31-31-57 snmp]#
Send a Test Trap
Now that we have our snmptrapd daemon running, let’s test it!
To send a test trap, run the following command:
snmptrap -v 2c -c public <destination_hostname> '' 220.127.116.11.4.1.8072.2.3.0.1 18.104.22.168.4.1.8072.2.3.2.1 i 123456
If you completed the installation and configuration step correctly, you should now see your trap in /var/log/snmptrapd.log (or wherever you have chosen to store the logs).
Tail -100 snmptrapd.log Agent_Address = 0.0.0.0 Agent_Hostname = UDP: [10.10.10.10]:23735->[10.10.31.83]:162 Date = 2018-08-17 16:23:10 Enterprise_OID = . Trap_Type = 0 Trap_SubType = 0 Community_Infosec_Context = TRAP2, SNMP v2c, community c4ct1 Uptime = 0 Description = Cold Start PDU_Attribute_Value_Pair_Array: DISMAN-EVENT-MIB::sysUpTimeInstance = 37:20:05:56.58 SNMPv2-MIB::snmpTrapOID.0 = NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate = 123456 ---
As you can see, our traps are translated and logged in a readable fashion. Splunk should have no problem handling them as key-value pairs.
Use the following configurations to instruct Splunk to monitor the file that snmptrapd is writing, and to parse and extract the key-value pairs that make up key information from the traps you have received.
It is recommended that you bundle these configuration files inside ITSI. Store them in $SPLUNK_HOME/etc/apps/SA-ITOA/local.
Note: Where you put the following configurations depends on your Splunk architecture. For more information, see About installing Splunk add-ons in the Splunk Add-ons Manual.
1. Configure inputs.conf
Add the following stanza to the local version of inputs.conf:
[monitor:///var/log/snmptrapd.log] disabled = false index = snmptrapd sourcetype = snmptrapd
This stanza tells Splunk where to look for our data. You only need to deploy this configuration on forwarders or Splunk instances co-located with snmptrapd.
2. Configure props.conf
Add the following stanza to the local version of props.conf:
[snmptrapd] DATETIME_CONFIG = KV_MODE = none LINE_BREAKER = ([\r\n]+)Agent_Address\s= MAX_TIMESTAMP_LOOKAHEAD = 30 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %Y-%m-%d %H:%M:%S TIME_PREFIX = Date\s=\s TZ = UTC category = Custom description = parse snmptrapd logging with custom kvpair splunk formatting disabled = false pulldown_type = true EXTRACT-node = ^[^\[\n]*\[(?P<node>[^\]]+) REPORT-snmptrapd = snmptrapd_kv
This stanza sets line breaking rules and extracts the timestamp. In this case, the timestamp is the time the snmptrapd server received the trap. It also ensures that Splunk knows how to parse the time format.
We perform field extractions, one being a ‘node’ field, the other pointing to a transform that will extract all key-value pairs in the trap.
Deploy this configuration to your heavy forwarder, indexer, and/or search head.
3. Configure transforms.conf
Add the following stanza to the local version of transforms.conf:
[snmptrapd_kv] DELIMS = "\n," = "
Deploy this configuration to your heavy forwarder, indexer, and/or search head.
4. Configure local.meta.conf
Add the following stanza to the local version of default.meta.conf:
 access = read : [ * ], write : [ admin ] export = system
This stanza includes a local.meta.conf file in the ITSI application, which exports permissions to the system. This ensures that ITSI can use our extractions. This setting is very important for ITSI correlation searches.
Deploy this configuration on your search heads.
5. Test the configuration in Splunk
If deployed successfully, you will see your traps in Splunk. Splunk should be extracting a ‘node’ field, and all key-value pairs in the traps:
Configure an ITSI Correlation Search
After you set up snmptrapd, configure how SNMP traps show up in ITSI Notable Events Review. Add the following correlation search in ITSI:
This simple correlation search brings in the raw snmptrapd data and presents it in Notable Events Review. It uses the ‘node’ and ‘Description’ fields for Titles/Descriptions and provides simple drilldown logic to the node.
Configure an ITSI Aggregation Policy
Use the same fields you used in the correlation search to configure the following aggregation policy in ITSI. The policy aggregates SNMP traps so they are displayed in a grouped format in Notable Events Review:
Test the Configuration
Return to ITSI Notable Events Review for a great initial look at the noise reduction you’ll see on your incoming SNMP data. The aggregated notable event group provides context and actionable data that your teams can easily manage.
From thousands of traps to a handful of actionable events!
After completing the steps outlined above, you can successfully use snmptrapd to receive SNMP traps and feed them into ITSI Notable Events Review. With the power of ITSI Event Analytics, you can use Splunk ITSI to its full potential to make your SNMP traps accessible, readable, and valuable to everyone.