Configure Splunk to pull a date out of a non-standard filename

Today I learned how to get Splunk to pull a date out of my non-standard filename. Did you know that, by default, Splunk can pull a date out of a filename like “20091202wow.file” ? What if you’ve got a filename like, “1202wow.file” ? Sorry, Splunk can’t do that by default — but you can configure it to do that by using a few tricks.

1) $SPLUNK_HOME/etc/datetime.xml

This is the datetime config file that ships with Splunk. It contains default rules used by Splunk to pull date/time out of your data. Obviously it can not cover ALL scenarios so we need to modify it when it does not seem to work properly.

2) First things, first – copy your $SPLUNK_HOME/etc/datetime.xml file to another location like $SPLUNK_HOME/etc/system/local/. Rename it if you like; or, keep it the same. Just make sure that you don’t edit the original file (just in case.)

3) We need to write a REGEX that will pull the month and day out of our filename and then integrate that REGEX into our custom datetime.xml file. You can use the existing configuration as an example. For brevity, here it the custom REGEX that will do this:

<define name="_masheddate3" extract="month, day">
<text><![CDATA[(?:^|source:).*?[^0-9]+(0\d|1[012])([012]\d|3[01])[^0-9]]]></text>
</define>

Edit your $SPLUNK_HOME/etc/system/local/datetime.xml file by adding somewhere in the upper portion of the datetime.xml file.

4) Reference your new REGEX name in the datePattern section towards the end of the datetime.xml:

<datePatterns>
...
<use name="_masheddate3"/>

5) Finally, use props.conf to refer Splunk to your custom datetime.xml for the intended sourcetype. Note that the path specified is relative to $SPLUNK_HOME:

[mysourcetype]
DATETIME_CONFIG = /etc/system/local/datetime.xml