IT

Splunk for DST

Splunk By Splunk October 27, 2010

Fellow Splunkers,

I hope everyone had a great International Caps Lock Day last Friday.  I KNOW I DID!!!1


As we approach that glorious time of the year when we all get an extra hour of sleep or drinking (I prefer to think of it as ‘time travel’), discussion around the topic of daylight savings time has centered on answering the following two questions:

  1. Are we (Splunk) testing for any DST-related bugs?
  2. Can we (Splunk) help detect machines that didn’t get the DST memo?

The answers are: “Of Course!” and “Obviously!”, respectively.  Let’s focus on the obvious.

The following search can help you identify hosts that are sending events “from the future“:

| metadata type=hosts | eval now = now() | where lastTime > now

  • The “metadata” command is most commonly used to provide the ‘hosts’, ‘sources’, and ‘sourcetypes’ list on the Splunk>Search app’s “Summary” dashboard.
  • “eval” is used to create a new field called ‘now’ with the current epoch time as the value.
  • “where” is used to compare two variables, in this case ‘lastTime’, a field created by ‘metadata’ that reflects the timestamp of the most last event received from the host.
    • Be careful not to confuse ‘lastTime’ with ‘recentTime’, which reflects the time on the receiving Splunk indexer that the last event was received from the host.
    • You can’t use ‘| search lastTime > now’ because search doesn’t compare two variables – it will treat the field to the right as a string literal.

There are some assumptions that I am making here:

  1. The clocks on most/all of your hosts are synced using NTP.  If you haven’t done this yet, stop reading this, spend an hour configuring ntp/ntpd (or better yet, delegate the task to someone else), and then resume reading.
  2. The clocks on your Splunk servers will correctly update with DST.
  3. Events on your hosts are not usually in the future.  If this is the case, you should stop reading, go fix it using either the TZ setting on your indexers’ props.conf or fix the drifting clocks wherever you find them.

Join in the discussion on answers if you have more insight, or post a topic on the community wiki if you feel like being charitable.

-araitz

----------------------------------------------------
Thanks!
Alex Raitz

Splunk
Posted by

Splunk