Lookup Before You Go-Go...Hunting

This is part one of the "Hunting with Splunk: The Basics" series.

Lookup before you go-go...hunting. (AKA How to use the lookup command for hunting.)

Often overlooked in the heat of the moment, the lookup command allows you to add csv files to Splunk and then run searches that match data in Splunk to the contents within that csv*. You can also use lookups to add context to your existing data in Splunk. For example, if you wanted to have additional context about a user in Splunk, you could import her office number or location as a lookup, and then use the lookup command to show that in Splunk. However, hunting—as always—is a slightly different use case than normal Splunking.

Let’s walk through some things you may want to hunt for in your environment.

Maybe you read a recent tweet that adversaries in your network often use public DNS servers. Perhaps you create a hypothesis that "Malicious actors and adversaries can be detected in our network by their use of public DNS servers.” Now, how would you find these public DNS servers in your network? A good place to start would be to get a list of public DNS servers and find hosts in your network that connects to these servers. To get started, let’s find a list of open/public DNS servers and download (or create) a csv.

Now that we have a csv, log in to Splunk, go to "Settings" > "Lookups" and click the “Add new” link for “Lookup Table Files”. You will see the window below. Click “Choose File” to upload your csv and assign a “Destination Filename” (in this case we kept it simple and called it “open_nameservers.csv”)

Click "Save."

Splunk returns you to the “Lookup Tables” menu. Click “Add New” in the “Lookup Definitions” to create a linkage between Splunk and the csv we just uploaded. In the example below, we used the (very imaginative) name “open_nameservers” for our definition name. This definition name will be used with the lookup command.

Now that we have our csv loaded and defined within Splunk, we can search for matches (e.g. any servers that responded to DNS servers that are in the lookup table).

Depending on the kind of data you have, your sourcetype may differ. In this example, we are leveraging Splunk Stream and gathering DNS data. The command

| lookup open_nameservers ip AS dest_ip

takes the field “ip” from the open_nameservers.csv that we defined and looks for matches in your data set in the field “dest_ip.” Whenever there is a match, Splunk adds additional context from the csv to the event.

We immediately see that a workstation in the network is connecting to 8.8.8[.]8 (a free public DNS server offered by Google) to resolve DNS domains. When the ip field in the csv and the dest_ip in the log matched, the name, country_id and city fields were added to the event because all three fields (name, country_id and city) are in the open_nameservers.csv.

Using this enrichment technique, we could then add another search command (ironically called “search”) to find all IP addresses that have a lookup match based on any of these new fields. In the example above, we searched the name field for any matches. If I was focused on looking for Google DNS servers, I could add:

| search name=*.google.com 

If we wanted to search for any IP addresses that were NOT public domain servers, we would simply swap = for !=

| search name!=*

This would display events from IP addresses that did not have a matching value in the field ‘name’ from the lookup file.

In the example above, we looked for IP addresses. However, this technique is not limited to IP addresses. In a previous blog post "Spotting the Adversary… with Splunk," I gave an example of Windows Event IDs that may indicate advanced adversary activity. This blog post provided an example list of Event IDs that could be searched for with OR statements. A more scalable solution would be to put those Event IDs into a lookup table.

In the example below, we create a csv with three fields:

As in the earlier example, we go to "Settings" > "Lookups" and click the “Add new” link for “Lookup Table Files.” The file is uploaded and assigned a name. The lookup definition is created and matched with the csv that was just imported. 

In the example below, the lookup definition and csv are called nsa_hunting. The eventcode field in nsa_hunting is matched to the field “EventCode” in the Windows event logs. This displays all of the event codes in sourcetype=WinEventLog:Security that match the NSA guidance for detecting Account and Group Activities by the adversary.

Lookup tables are a very powerful tool to match IOCs and strings. However, there are a few things to keep in mind:

• If you want to add entries to your csv that contain wildcards (i.e.  *maliciousioc.com), you will need to edit the transforms.conf. You cannot do this via the GUI.
• Make sure you create your csv correctly. Sometimes, Excel may not create a file correctly and Splunk can’t ingest it correctly. If Splunk fails to ingest, look for special characters hidden in the .csv.
• I find it very useful to always have a second field in my lookup. This second field is something that you can search for to verify that Splunk is finding your match.  If it doesn’t exist, you never had any matches.
• Lookups can be kept up to date in a few ways—two popular methods are to upload a new .csv to the Splunk search head, another is to use the “outputlookup” SPL command to take the events from a Splunk search and add those to a lookup; these kinds of searches can be scheduled so that you always have a fresh lookup.

With that, I leave you with thoughts of IOCs from Mandiant reports dancing in your head. Happy Hunting. :-)

(*Note: Of course, lookups don’t always have to be csv files—they can be dynamic commands, they can pull data from databases, and they can leverage Splunk’s KV store.)