By Splunk October 24, 2008
A customer asked about a complicated search that could be vastly simplified by using inputcsv to input a list of values from a file, a feature added for 3.3.x. It’s documented as an internal search command here:
http://www.splunk.com/doc/latest/user/UnsupportedCommands#inputcsv
We are talking about promoting it to public, so while it says unsupported it does work. Here’s how:
I’ve got events from my webserver for my new domain and I want to see what real hits it’s getting and not my own. They look like this:
66.249.70.86 - - [23/Oct/2008:01:42:21 -0700] "GET /category/admin/ HTTP/1.1" 200 5158 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
And I’ve gotten some traffic already:
$ ./splunk dispatch 'source=/var/log/apache2/mynewdomain_access_log | stats count'
count
-----
11424
It’s a standard format that was automatically recognized as sourcetype access_common, so the extracted field “clientip” is already there. I create a csv file containing the values I want to exclude like this:
clientip
xxx.xxx.xxx.xxx
yyy.yyy.yyy.yyy
zzz.zzz.zzz.zzz
This file needs to exist relative to $SPLUNK_HOME/var/run/splunk, so to avoid specifying a path in my search I’ll just put it there. Note that I could also have used xxx.xxx.xxx.* if I wanted to, wildcards are ok.
Now I can do this search:
./splunk dispatch 'source=/var/log/apache2/mynewdomain_access_log NOT [inputcsv mycsvfile.csv]'
$ ./splunk dispatch 'source=/var/log/apache2/myghettodatacenter_access_log NOT [inputcsv mycsvfile.csv] | stats count'
count
-----
121
and only get the ones that aren’t from my network. This search also works from the UI as
source="/var/log/apache2/mynewdomain_access_log" NOT [inputcsv mycsvfile.csv]