Configure timestamp recognition

Configure timestamp recognition

Splunk uses timestamps to correlate events by time, create the histogram in Splunk Web and to set time ranges for searches. Timestamps are assigned to events at index time.

Splunk assigns a timestamp to most events based on information in the raw event data. If an event doesn't contain timestamp information, Splunk attempts to assign a timestamp value to the event as it's indexed. Splunk stores timestamp values in the _time field (in UTC time format).

Most events don't require any special timestamp handling; you can just let Splunk handle it without any configuration.

Precedence rules for timestamp assignment

Splunk uses the following precedence to assign timestamps to events:

1. Look for a time or date in the event itself using an explicit TIME_FORMAT if provided.

Use positional timestamp extraction for events that have more than one timestamp value in the raw data.

2. If no TIME_FORMAT is provided, or no match is found, attempt to automatically identify a time or date in the event itself.

3. If an event doesn't have a time or date, use the timestamp from the most recent previous event of the same source.

4. If no events in a source have a time or date, look in the source (or file) name.

5. For file sources, if no time or date can be identified in the file name, use the modification time on the file.

6. If no other timestamp is found, set the timestamp to the current system time (the time at which the event is indexed by Splunk).

Configure timestamps

Most events don't require any special timestamp handling; you can just let Splunk handle it without any configuration.

For some sources and distributed deployments, you may have to configure timestamp formatting to extract timestamps from events. Configure Splunk's timestamp extraction processor by editing props.conf.

Configure how Splunk recognizes timestamps by editing props.conf. Splunk uses strptime() formatting to identify timestamp values in your events. Specify what Splunk recognizes as a timestamp by setting a strptime() format in the TIME_FORMAT= key.

Note: If your event has more than one timestamp, set Splunk to recognize the correct timestamp with positional timestamp extraction.

Use $SPLUNK_HOME/etc/system/README/props.conf.example as an example, or create your own props.conf. Make any configuration changes to a copy of props.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

Configure any of the following attributes in props.conf to set Splunk's timestamp recognition. Refer to $SPLUNK_HOME/etc/system/README/props.conf.spec for full specification of the keys.

[<spec>]
DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>
MAX_TIMESTAMP_LOOKAHEAD = <integer>
TIME_PREFIX = <regular expression>
TIME_FORMAT = <strptime-style format>
TZ = <posix timezone string>
MAX_DAYS_AGO = <integer>
MAX_DAYS_HENCE = <integer>

[<spec>]

• <spec> indicates what to apply timestamp extraction to. This can be one of the following:
  • <sourcetype>, the sourcetype of an event.
  • host::<host>, where <host> is the host of an event.
  • source::<source>, where <source> is the source of an event.
• If an event contains data that matches the value of <spec>, then the timestamp rules specified in the stanza apply to that event.
• Add additional stanzas to customize timestamp recognition for any type of event.

DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>

• Specify a file to use to configure Splunk's timestamp processor (by default Splunk uses $SPLUNK_HOME/etc/datetime.xml).
• To use a custom datetime.xml, specify the correct path to your custom file in all keys that refer to datetime.xml.
• Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running.
• Set DATETIME_CONFIG = CURRENT to assign the current system time to each event as it's indexed.

MAX_TIMESTAMP_LOOKAHEAD = <integer>

• Specify how far (how many characters) into an event Splunk should look for a timestamp.
• Default is 150 characters.
• Set to 0 to assign current system time at an event's index time.

TIME_PREFIX = <regular expression>

• Use a regular expression that points to the space exactly before your event's timestamp.
  • For example, if your timestamp follows the phrase Time=, your regular expression should match this part of the event.
• The timestamp processor only looks for a timestamp after the TIME_PREFIX in an event.
• Default is none (empty).

TIME_FORMAT = <strptime-style format>

• Specify a strptime() format string to extract the date.
• Set strptime() values in the order that matches the order of the elements in the timestamp you want to extract.
• Splunk's timestamp processor starts processing TIME_FORMAT immediately after a matching TIME_PREFIX value.
• Doesn't support in-event timezones.
• TIME_FORMAT starts reading after a matching TIME_PREFIX.
• The <strptime-style format> value must contain the hour, minute, month, and day.
• Default is empty.

TZ = <timezone string>

• Specify a time-zone setting using a value from the zoneinfo TZID database.
• For more details and examples learn how to configure timezone offsets.
• Default is empty.

MAX_DAYS_AGO = <integer>

• Specify the maximum number of days in the past (from the current date) for an extracted date to be valid.
• For example, if MAX_DAYS_AGO = 10 then dates that are older than 10 days ago are ignored.
• Default is 1000.

Note: You must configure this setting if your data is more than 1000 days old.

MAX_DAYS_HENCE = <integer>

• Specify the maximum number of days in the future (from the current date) for an extracted date to be valid.
• For example, if MAX_DAYS_HENCE = 3 then dates that are more than 3 days in the future are ignored.
• The default value (2) allows dates that are tomorrow.

Note: If your machines have the wrong date set or are in a timezone that is one day ahead, set this value to at least 3.

Enhanced strptime() support

Configure timestamp parsing in props.conf with the TIME_FORMAT= key. Splunk implements an enhanced version of Unix strptime() that supports additional formats (allowing for microsecond, millisecond, any time width format, and some additional time formats for compatibility). See the table below for a list of the additionally supported strptime() formats.

In previous versions, Splunk parsed timestamps using only the standard Linux strptime() conversion specifications. Now, in addition to standard Unix strptime() formats, Splunk's strptime() implementation supports recognition of the following date-time formats:

%N	For GNU date-time nanoseconds. Specify any sub-second parsing by providing the width: %3N = milliseconds, %6N = microseconds, %9N = nanoseconds.
%Q,%q	For milliseconds, microseconds for Apache Tomcat. %Q and %q can format any time resolution if the width is specified.
%I	For hours on a 12-hour clock format. If %I appears after %S or %s (like "%H:%M:%S.%l") it takes on the log4cpp meaning of milliseconds.
%+	For standard UNIX date format timestamps.
%v	For BSD and OSX standard date format.
%z, %::z, %:::z	GNU libc support.
%o	For AIX timestamp support (%o used as an alias for %Y).
%p	The locale's equivalent of AM or PM. (Note: there may be none.)

strptime() format expression examples

Below are some sample date formats with strptime() expressions that handle them.

1998-12-31	%Y-%m-%d
98-12-31	%y-%m-%d
1998 years, 312 days	%Y years, %j days
Jan 24, 2003	%b %d, %Y
January 24, 2003	%B %d, %Y
q|25 Feb '03 = 2003-02-25|	q|%d %b '%y = %Y-%m-%d|

Examples

Your data might contain an easily recognizable timestamp to extract such as:

...FOR: 04/24/07 PAGE 01...

The entry in props.conf is:

[host::foo]
TIME_PREFIX = FOR: 
TIME_FORMAT = %m/%d/%y

Your data might contain other information that Splunk parses as timestamps, for example:

...1989/12/31 16:00:00 ed May 23 15:40:21 2007...

Splunk extracts the date as Dec 31, 1989, which is not useful. In this case, configure props.conf to extract the correct timestamp from events from host::foo:

[host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y

This configuration assumes that all timestamps from host::foo are in the same format. Configure your props.conf stanza to be as granular as possible to avoid potential timestamping errors.

You can also configure Splunk's timestamp extraction processor to:

• Apply timezone offsets.
• Recognize European date format.
• Pull the correct timestamp from events with more than one timestamp.
• Improve indexing performance.

Finally, train Splunk to recognize new timestamp formats.