By Guest September 10, 2015
This is a guest post contributed by Luke Netto, Security Engineer, Level 3 Communications
Lookups are a great way to enrich events with more meaningful data, however they are a very costly operation to search events using the enhanced fields.
While working on a recent project as a security engineer at Level 3 Communications, I found a much more efficient way to search these new fields. This trick works with any lookups that can be used as a reverse lookup. In the example below I allow users to search logs containing integer-formatted IP addresses using dot-decimal notation without performing a lookup on each event.
In order to search for 141.146.8.66 in this dataset in the most efficient way possible you would have to search for the integer value 2375157826 (figure 1 & 2). One could perform a lookup and convert all the integer IP addresses to dot-decimal and then search, but it is very costly and takes 10x the time to run. (figure 3 & 4). Using the search job inspector, we can see the second search required Splunk to scan all 455,544 events instead of just the 40,064 events that contained the value 2375157826 (figures 2 & 4).
Figure 1
Figure 2
Figure 3
Figure 4
Converting the IP address to integer format is not a huge task if only searching for one or two addresses but what if you created a dashboard (figure 5) and needed it to perform efficiently?
Figure 5
Before discovering this trick I would have used the search in figure 3 to power this dashboard and allow filtering by IP address. What if there was a way we could perform the lookup on the user input and convert the dot-decimal notation to integer format before Splunk executes the search? What if we could optimize our search query if the user searches for a single IP address versus a subnet or CIDR block? We can!
I created a scripted lookup that returned a different string based on the user input. If the user enters a “*” the script returns the same value. If the user entered a single dot-decimal IP address the script returns the integer format. If the user entered a subnet in CIDR or glob notation the script returns the starting and ending IP address in integer format. If the user enters an invalidate input the script returns “NOT *” and prevents the dashboard from loading.
Now how do we actually use this? Instead of having the script simple return values, we can have it return SPL syntax! If the user is searching for a single IP address the ideal search would be “src_addr=X OR dest_addr=X”, where X is the integer formatted IP address (figure 6).
If the user is searching for a range one possible search is “(src_addr >= X AND src_addr <= Y) OR (dest_addr >= X AND dest_addr <= Y)”, where X is the starting IP address and Y is the ending IP address in integer format (figure 7).
Figure 6
Figure 7
Let’s break apart what is happening here. The command “| gentimes start=-1” allows us to run a lookup without Splunk actually returning any data. We use eval to set the user input to a variable and return “output” from the lookup function. The last command “return $range” tells Splunk we only want value of the field “range”, not the field and value.
Using this creates a search that scans only 46,064 events like the search in figure 1 and 2 (figure 8).
Figure 8
Figure 9
Figure 10
Figure 11
Thanks,
Luke Netto
Level 3 Communications