Dont forget to index your config files!

Dont forget to index your config files!
Why?
Because splunk is a great way to track changes and see differences in your configs.
For most troubleshooting and compliance situations having a historical recored of all your configurations just goes hand in hand with the log data. They are two sides of the same coin.

The cool thing is that it takes just a few seconds to get up and running. If you have splunk installed its all but free to index your configs – they are small in size compared to log files. Even if you indexed all configs in a 2000 machine deployment it would not come close to the volume of even a small size proxy log.

30 second refresher:
Just tail /etc you will capture most of the interesting configs on your box.

from the cli:
> splunk add tail /etc

or in UI just add a tail to /etc

Thats it. That is all you need to do.

** note ** you should grab 3.1 ( http://download.splunk.com ) as there were some bugs in 3.0’s config processing.

Splunk does something very interesting/cool with config files ( also source and script files ).
Unlike log files, splunk will treat the entire contents of the config files as a single result. This means that if i were to tail the file syslog.conf I would immediately get one result when searching for a key such as “authpriv” or for source=syslog.conf. This one result will be the contents of the file with the timestamp of the last mod time.

But as soon as the file changes splunk will recognize the change and re-index the entire file as another single result. Now there will be two results for the search “authpriv” or when searching for source=syslog.conf. At this time, splunk will have effectively kept 2 “copies” of the config file. Over time, as each change is made, splunk will accumulate more and more results for that config. What is interesting is that over a period of weeks you will have all copies of syslog.conf or all conf files in /etc for that matter. This behavior is very different from that of most search engines where they only keep the last version of the document. You can search for configs by keyword, timestamp, attribute, or report/graph changes. Overtime Splunk will keep a complete historical record of all versions of your configs – to go along with all your logs ;-).

A couple things to try after you index your configs:

1) See all config changes by time ( you will need to have splunk running to accumuate anything interesting )

Search for "sourcetype::config_file" – you should see the following type of result/graph which is showing all config file changes for the last day. Notice that my configs are changing all the time – during the 10AM hour my configs changed 35 times… and i did not edit one of them!

2) See top for config files:
Clicking on the report link and choosing “source” from the left hand side I get the following nice top graph. Two clicks and you get a nice graph 😉

3) See changes for a single config:
In this case I have picked just a single source by clicking on the sources drop-down and picking “/etc/printcap” – alternately you could just search for source=/etc/printcap Notice how often throughout the weekday this file is changing – while at home on Sat/Sun printcap did not change. This is running on my laptop so as i move from floor to floor here at splunk the network changes and configs change.

4) Diff’ing two results:
It’s not really enough to just see when a file changes but it really helps to be able to see the differences between any two.
Splunk has a rudimentary ( soon to get better ) diff search processor.
To see differences between two versions of a file just add | diff to the end of your search.
The following screen shot shows piping the search source=/etc/printcap through | diff

Diff Options:
There are a few options you can supply to the diff command. (*** NOTE – i just noticed that the type ahead is wrong of the diff command. I’ll make sure its fixed in next maint release)

• – position1 & position2=[int] : you can pick which two results to diff
• – context=[true | false] : context diff will show the changes in the context of the differences. Try it see if it makes sense.
• – header=[true | false] : optionally you can show a header that tries to explain the diff output
• – attribute=[attribute name] : you can choose to diff just a single attribute of the reults.

If you want to add your own configs outside of etc just tail the files in splunk as usual and be sure to set the sourcetype to config_file

If you’re up to it, the code for the diff command written in python and is located in SPLUNK_HOME/etc/searchscripts/diff.py.
Feel free to add/change and otherwise improve the command.
Future releases will do more to help visualize differences.

Like i said, It could not be more easy!

In our next release you will see way more cool config and source code support.
In the mean time, eat your configs and let us know if you have any idea’s on how to build out configuration file features.

e.