By Splunk October 14, 2013
Quite a while ago I wrote a blog post entitled The Splunk App for Active Directory and How I tamed the Security Log. It detailed how to limit the amount of data that was going into the Splunk index through filtering. I included two techniques – firstly, filtering by event code so that you didn’t include the events you didn’t want; and secondly, filtering the explanatory text on the end of each event. Splunk 6 makes this so much easier that the prior blog post is not even relevant any more.
Let’s say you don’t want firewall events. From the previous blog post, event ID 5156 and 5157 detail the firewall connection accept and deny messages. Let’s say those are not relevant to us. Previously, we had to add a props.conf stanza to initiate a filtering action that was done in transforms.conf – it was complicated. In Splunk 6, everything is done in inputs.conf. Here is a new inputs.conf stanza for you:
[WinEventLog:Security] disabled = false blacklist = 5156-5157
There are two new parameters you can specify – the first, shown here, is a black list of all the event IDs you don’t want to monitor. You can use ranges (as I did here), or comma-separate the event IDs or event comma-separate ranges of event IDs. The second parameter is a whitelist – if you have more that you don’t want to keep than you want to keep. It follows the same format.
The second facility I wrote about was suppressing the explanatory text. Splunk 6 makes this easier as well. Let’s take a look at a typical windows event prior to the text suppression:
10/14/2013 08:29:33 AM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4634 EventType=0 Type=Information ComputerName=SP-SQL.bd.splunk.com TaskCategory=Logoff OpCode=Info RecordNumber=3544 Keywords=Audit Success Message=An account was logged off. Subject: Security ID: BD\a-ahall Account Name: a-ahall Account Domain: BD Logon ID: 0x5886A Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
You see that “This event is generated…” text – that’s the explanatory text. It’s the same for every single event. Since these events get generated every 10-15 minutes for every single user on your domain controllers and they are 100+ bytes, you can see how they can add up. And that’s just one example. Every single security event has similar explanatory text. In Splunk 6, you can add a new parameter to your inputs.conf stanza to supress the Message field:
[WinEventLog:Security] disabled = 0 suppress_text = 1
Now when you get those events, this is what they look like:
10/14/2013 08:43:07 AM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4672 EventType=0 Type=Information ComputerName=SP-SQL.bd.splunk.com TaskCategory=Special Logon OpCode=Info RecordNumber=3546 Keywords=Audit Success Message=
You will note that is NO message text at all. This is fine for some logs (usually custom service logs) where the message is not important. However, you will still need to use the same transform as before if you want some of the message but not all – for example, with the Security log. Since all the stanzas of the same name are munged together, you should be careful about setting the suppress_text parameter. In particular, do not set the suppress_text parameter on WinEventLog:Security as it will not log any of the important contextual information within the security log.
These two changes can make your windows event log gathering more efficient, but as always – be careful of what you throw away.