Configure timestamp extraction

Configure timestamp extraction

Configure how Splunk recognizes timestamps by editing props.conf. Splunk uses strptime() formatting to identify timestamp values in your events. Specify what Splunk recognizes as a timestamp by setting a strptime() format in the TIME_FORMAT= key.

Learn about Splunk's enhanced strptime() format support.

Note: If your event has more than one timestamp, set Splunk to recognize the correct timestamp with positional timestamp extraction.

Configure timestamp extraction in props.conf

Use $SPLUNK_HOME/etc/bundles/README/props.conf.example as an example, or create your own props.conf. Make any configuration changes to a copy of props.conf in $SPLUNK_HOME/etc/bundles/local/, or your own custom bundle directory. For more information on configuration files in general, see how configuration files work.

Configure any of the following attributes in props.conf to set Splunk's timestamp recognition. Refer to $SPLUNK_HOME/etc/bundles/README/props.conf.spec for full specification of the keys.

[<spec>]
DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>
MAX_TIMESTAMP_LOOKAHEAD = <integer>
TIME_PREFIX = <regular expression>
TIME_FORMAT = <strptime-style format>
TZ = <posix timezone string>
MAX_DAYS_AGO = <integer>
MAX_DAYS_HENCE = <integer>

[<spec>]

• <spec> indicates what to apply timestamp extraction to. Can be one of the following:
  • <sourcetype>, the sourcetype of an event.
  • host::<host>, where <host> is the host of an event.
  • source::<source>, where <source> is the source of an event.
• If an event contains data that matches the value of <spec>, then the timestamp rules specified in the stanza apply to that event.
• Add additional stanzas to customize timestamp recognition for any type of event.

DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>

• Specify a file to use to configure Splunk's timestamp processor (by default Splunk uses $SPLUNK_HOME/etc/datetime.xml).
• To use a custom datetime.xml, specify the correct path to your custom file in all keys that refer to datetime.xml.
• Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running.
• Set DATETIME_CONFIG = CURRENT to assign current system time to each event as it's indexed.

MAX_TIMESTAMP_LOOKAHEAD = <integer>

• Specify how far (how many characters) into an event Splunk should look for a timestamp.
• Default is 150 characters.
• Set to 0 to assign current system time at an event's index time.

TIME_PREFIX = <regular expression>

• Use a regular expression that points to the space exactly before your event's timestamp.
  • For example, if your timestamp follows the phrase Time=, your regular expression should match this part of the event.
• The timestamp processor only looks for a timestamp after the TIME_PREFIX in an event.
• Default is none (empty).

TIME_FORMAT = <strptime-style format>

• Specify a strptime() format string to extract the date.
• Set strptime() values in the order that matches the order of the elements in the timestamp you want to extract.
• Splunk's timestamp processor starts processing TIME_FORMAT immediately after a matching TIME_PREFIX value.
• Doesn't support in-event timezones.
• TIME_FORMAT starts reading after a matching TIME_PREFIX.
• The <strptime-style format> value must contain the hour, minute, month, and day.
• Default is empty.
• Learn what strptime() formats Splunk supports.

TZ = <timezone string>

• Specify a time-zone setting using a value from the zoneinfo TZID database.
• For more details and examples learn how to configure timezone offsets.
• Default is empty.

MAX_DAYS_AGO = <integer>

• Specify the maximum number of days in the past (from the current date) for an extracted date to be valid.
• For example, if MAX_DAYS_AGO = 10 then dates that are older than 10 days ago are ignored.
• Default is 1000.

Note: You must configure this setting if your data is more than 1000 days old.

MAX_DAYS_HENCE = <integer>

• Specify the maximum number of days in the future (from the current date) for an extracted date to be valid.
• For example, if MAX_DAYS_HENCE = 3 then dates that are more than 3 days in the future are ignored.
• The default value (2) allows dates that are tomorrow.

Note: If your machines have the wrong date set or are in a timezone that is one day ahead, set this value to at least 3.

Enhanced strptime() support

Configure timestamp parsing in props.conf with the TIME_FORMAT= key. Splunk implements an enhanced version of Unix strptime() that supports additional formats (allowing for microsecond, millisecond, any time width format, and some additional time formats for compatibility). See the table below for a list of the additionally supported strptime() formats.

In previous versions, Splunk parsed timestamps using only the standard Linux strptime() conversion specifications. Now, in addition to standard Unix strptime() formats, Splunk's strptime() implementation supports recognition of the following date-time formats:

%N	For GNU date-time nanoseconds. Specify any sub-second parsing by providing the width: %.3N = milliseconds, %.6N = microseconds, %.9N = nanoseconds.
%Q,%q	For milliseconds, microseconds for Apache Tomcat. %Q and %q can format any time resolution if the width is specified.
%I	For hours on a 12-hour clock format. If %I appears after %S or %s (like "%H:%M:%S.%l") it takes on the log4cpp meaning of milliseconds.
%+	For standard UNIX date format timestamps.
%v	For BSD and OSX standard date format.
%z, %::z, %:::z	GNU libc support.
%o	For AIX timestamp support (%o used as an alias for %Y).
%p	The locale's equivalent of AM or PM. (Note: there may be none.)

strptime() format expression examples

Below are some sample date formats with strptime() expressions that handle them.

1998-12-31	%Y-%m-%d
98-12-31	%y-%m-%d
1998 years, 312 days	%Y years, %j days
Jan 24, 2003	%b %d, %Y
January 24, 2003	%B %d, %Y
q|25 Feb '03 = 2003-02-25|	q|%d %b '%y = %Y-%m-%d|

Examples

Your data might contain an easily recognizable timestamp to extract such as:

...FOR: 04/24/07 PAGE 01...

The entry in props.conf is:

[host::foo]
TIME_PREFIX = FOR: 
TIME_FORMAT = %m/%d/%y

Your data might contain other information that Splunk parses as timestamps, for example:

...1989/12/31 16:00:00 ed May 23 15:40:21 2007...

Splunk extracts the date as Dec 31, 1989, which is not useful. In this case, configure props.conf to extract the correct timestamp from events from host::foo:

[host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y

This configuration assumes that all timestamps from host::foo are in the same format. Configure your props.conf stanza to be as granular as possible to avoid potential timestamping errors.