By Splunk March 10, 2014
Threat intel feeds are a good way to add security context to your Splunk data with IP addresses, domain/host names or files. These feeds are generally accessible via some manner of web requests. Splunk Enterprise Security App has a Unified Threat Management framework for integrating threat intelligence feeds that makes these integrations easy . If the threat content you need to use is easy to download, you should be able to simply use the Configure -> Data Enrichment -> Threat Lists -> New form in the ES product.
But sometimes, a feed provider may require a number of steps before we can get the actual feed. Here’s how to handle a more difficult integration easily, using Symantec DeepSight’s threat feed as an example. The code bundle for this app is available on Splunk Apps.
Integrating the Symantec DeepSight Feed into Splunk Enterprise via lookups
- Get Data into Splunk; in this case we are working with a scripted input, but DB Connect or a modular input may also be good choices depending on the task at hand.
- Create a script to download the feed from Symantec Deep Sight portal: $SPLUNK_HOME/etc/TA-Symantec-DeepSight/bin/DSDownload.py
- Here’s some example DeepSight output. It’s good to know that the Enterprise Security Threat Framework expects IP addresses without quotes; this app’s DSDownload.py script handles that. But if your feed uses special characters for IP addresses, you will have to account for that.
ip,consecutive_listings,listing_ratio,reputation,behaviour,hostility,confidence
"8.13.91.100",25,76,10,"attack",5,3
"8.13.91.100",25,76,10,"malware",5,1
"8.15.25.59",90,90,10,"spam",5,4
"8.15.25.59",90,90,10,"CnC",,4
NOTE: This is an example only.- Edit the app’s setup.xml to securely store the credentials while the script is not running: $SPLUNK_HOME/etc/apps/TA-Symantec-DeepSight/default/setup.xml. Note that this credential can be managed in Enterprise Security via Configure -> Credential Manager; you can also create credentials there if you already have ES.
- Write out IP and description and action as columns in a lookup file
- Edit transforms.conf to define the lookup table at
- $SPLUNK_HOME/etc/TA-Symantec-DeepSight/default/tranforms.conf
- Edit props.conf to apply the lookup table at $SPLUNK_HOME/etc/TA-Symantec_DeepSight/default/props.conf
- Create a lookup directory at $SPLUNK_HOME/etc/TA-Symantec_DeepSight/lookups
- Test the lookup in Splunk Enterprise
| inputlookup deepSightIpFeed - Tell ES’s Unified Threat Management framework how to eat the lookup with Manage -> Data Inputs -> Threat Lists.
- Gathering a lookup instead of a website has already been configured, so we can copy part of this configuration. Find local threatlist and click “clone”, then modify the fields column to say
- Alternatively, you can make a new entry directly in your $SPLUNK_HOME/etc/TA-Symantec_DeepSight/default/inputs.conf:
ip:$1,category:$5,risk:$7,description:symantec_threatlist[threatlist://symantec_threatlist_ip_reputation_feed]
delim_regex = ,
description = Symantec Deepsight IP Reputation
disabled = false
fields = ip:$1,category:$5,risk:$7,description:symantec_threatlist
type = Symantec Deepsight IP Reputation
url = lookup://deepSightIpFeed
- At Manage -> Data Inputs -> Threat Lists, click disable and enable for the new Symantec DeepSight IP Reputation feed to initiate a threatlist aggregation. You can verify that it’s working properly by running this search:
index=_internal sourcetype="python_modular_input" category=threatlist name=symantec_threatlist_ip_reputation_feed
The overall goal is to have an automated lookup that is populated/updated by connecting to the Deep Sight portal. The feeds are updated daily by Symantec but we don’t know exactly when that happens, so we’ll check twice a day.
Our approach used a script to acquire the feed. We could have used a modular input as well. More details on modular inputs can be found here: Modular Inputs Introduction
Thanks to Monzy and James for their help!
----------------------------------------------------
Thanks!
Jack Coates