Create indexed fields via configuration files

Create indexed fields via configuration files

Splunk automatically adds indexed fields such as host, source, source type, event type, etc. Create your own custom indexed fields. Once you have created a new indexed field, it appears in the Fields drop-down menu in Splunk Web. You can also search on it, as well, by typing $CUSTOM_FIELD=foo in your search.

Note: Indexed fields have performance implications. Read about how fields work for more information. It is rarely necessary to create indexed fields (versus extracted fields). You may want to use indexed fields if you search for expressions like foo!="bar" or NOT foo="bar" and the field foo nearly always takes on the value bar. Another common reason to use indexed fields is if the value of the field exists outside of the field more often than not. For example, if you commonly search for foo="1", but 1 occurs in many events that do not have foo="1", you may want to index foo.

Configuration

Define additional indexed fields by editing props.conf, transforms.conf and fields.conf.

Edit these files in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.

transforms.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/transforms.conf:

[<unique_stanza_name>]
REGEX = <your_regex>
FORMAT = <your_custom_field_name>::"$1"
WRITE_META = true

• $UNIQUE_STANZA_NAME = name your stanza. Use this name later to configure props.conf.
• REGEX = create a regex that recognizes your custom field value.
• FORMAT = inserts <your_custom_field_name> before the value you've extracted via regex as $1.
  • In order to properly display field values containing whitespace in Splunk Web, apply quotes to the FORMAT key.
  • FORMAT = <your_custom_field_name>::"$1"
  • Multiple fields can be extracted using a single regex that contains multiple match groups
  • FORMAT = <your_first_field>::"$1" <your_second_field>::"$2"
• WRITE_META = set this to true to write your field name and value to meta. This is where indexed fields are stored.

props.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/props.conf:

[<spec>]
TRANSFORMS-<value> = <unique_stanza_name>

• <spec> can be:
  • <sourcetype>, the sourcetype of an event.
  • host=<host>, where <host> is the host for an event.
  • source=<source>, where <source> is the source for an event.
• <unique_stanza_name> is the name of your stanza from transforms.conf.
• <value> is any value you want to give to your stanza to identify its name-space.

fields.conf

Add an entry to fields.conf for your new indexed field.

[<your_custom_field_name>]
INDEXED=true

• <your_custom_field_name> is the name of the custom field you set in transforms.conf.
• Set INDEXED=true to indicate that the field is indexed.
• If a field of the same name is extracted in other data (rather than indexed), you must not set INDEXED=true. In this case, you must also set INDEXED_VALUE=false if events exists that have values of that field which are not indexed. An example of this case would be a regex like: A(\d+)B, where the string A1234B would yield the value 1234 for the field, but the event cannot be found by searching for 1234.

Examples

Example 1

This example creates an indexed field called err_code.

transforms.conf

In $SPLUNK_HOME/etc/system/local/transforms.conf add:

[netscreen-error]
REGEX =  device_id=[^ ]+\s+\[w+\](.*)(?
FORMAT = err_code::"$1"
WRITE_META = true

This stanza takes 'device_id=' followed with a word within brackets and a text string terminating with a colon. The source type of the events is testlog.

Comments:

• The FORMAT = line contains the following values:
  • err_code:: is the name of the field.
  • $1 refers to the new field written to the index. It is the value extracted by REGEX.
• WRITE_META = true is an instruction to write the content of FORMAT to the index.

props.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/props.conf:

[testlog]
TRANSFORMS-netscreen = netscreen-error

fields.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/fields.conf:

[err_code]
INDEXED=true

Example 2

This example creates two indexed fields called username and login_result.

transforms.conf

[ftpd-login]
REGEX = Attempt to login by user: (.*): login (.*)\.
FORMAT = username::"$1" login_result::"$2"
WRITE_META = true

This stanza finds the literal text Attempt to login by user: , extracts a username, followed by a colon, and then the result, which is followed by a period. A line might look like

2008-10-30 14:15:21 mightyhost awesomeftpd INFO Attempt to login by user: root: login FAILED.

props.conf

[ftpd-log]
TRANSFORMS-login = ftpd-login

fields.conf

[username]
INDEXED=true
[login_result]
INDEXED=true

How indexed fields work in detail

Splunk builds indexed fields by writing to _meta. Here's how it works:

• _meta is modified by all matching transform that contain either DEST_KEY = meta or WRITE_META = true.
• Each transform can overwrite _meta, so use WRITE_META = true to append _meta.
  • If you don't use WRITE_META, then start your FORMAT with $0.
• After _meta is fully built during parsing, the text is interpreted in the following way.
  • The text is broken into units; each unit is separated by whitespace.
  • Quotation marks (" ") group characters into larger units, regardless of whitespace.
  • Backslashes ( \ ) immediately preceding quotation marks disable the grouping properties of quotation marks.
  • Backslashes preceding a backslash disable that backslash.
  • Units of text that contain a double colon (::) are turned into extracted field. The text on the left side of the double colon becomes the field name, and the right side becomes the value.

Note: Indexed fields with regex-extracted values containing quotation marks will generally not work, and backslashes may also have problems. Extracted fields do not have these limitations.

Quoting example

WRITE_META = true

FORMAT = field1::value field2::"value 2" field3::"a field with a \" quotation mark" field4::"a field which ends with a backslash\\"