How fields work

A field is any searchable name/value pair. A field is distinguished from the free-form indexed segments of an event in that fields are labeled and can be searched by label. For example, host=foo is a field whose name is host and value is foo. Search on any field name or specific value of a field.

Search time vs indexed time

Fields are categorized by how and when they are processed. Configure fields that are created at indexing time (indexed fields) or fields that are created at search time (extracted fields).

Indexed fields are captured in real time as events are processed and indexed by Splunk. Information on where the event came from, what type of event, source type, etc, are extracted by Splunk's input processor. Configure Splunk to index new indexed fields. Indexed fields are useful for creating event type templates.

Extracted fields are created at search time. Splunk picks out obvious name/value pairs in search results, such as user_id or client_ip. This dynamic extracted field list can be used in filters and reports. Configure Splunk to create new extracted fields. Extracted fields cannot be used to create new event types.

Performance implications

Fields extracted at index time have a negative impact on indexing performance. However, they are quicker to return in a search (especially with fields.conf configured correctly). Indexed fields are less flexible, however, and if you want to change fields for your data, your only option is to re-index the entire dataset.

Extracted fields are much more flexible and can be redefined on the fly. They have no negative impact on indexing. However, they do make searches run slower.

If you are not completely sure which option to choose, try extracted fields first. They allow more flexibility, should you make a mistake with your configuration.

Configure fields

Define fields with Splunk's interactive field extraction. Or create fields using props.conf, transforms.conf and fields.conf using the following process. Note: The process for defining new fields is nearly the same for both indexed and extracted fields.

Determine a pattern to identify the field in the event.
Write a regular expression to extract the field from the event.
Edit your custom props.conf and transforms.conf files. (Note: DO NOT edit the copy in $SPLUNK_HOME/etc/bundles/default/.)
In props.conf, specify either the source, source type or host containing the events and assign a name to identify the transform in transforms.conf.
In transforms.conf, create the named transform stanza, and supply the regex to extract the field.
For indexed fields, write this information to the index.
Configure fields.conf to set additional processing information. Read more about how to configure fields.conf.

Disable automatically extracted fields

Splunk automatically extracts fields from your data and adds them to the Fields drop-down menu in Splunk Web. Disable this feature via props.conf. You can turn off extracted fields for a specific source, sourcetype or host. Add the attribute/value pair KV_MODE = none for the appropriate [<spec>] in $SPLUNK_HOME/etc/bundles/local/props.conf:

[<spec>]
KV_MODE = none

<spec> can be:
1. <sourcetype>, the sourcetype of an event
2. host::<host>, where <host> is the host for an event
3. source::<source>, where <source> is the source for an event

Configuration files for fields

Indexed and extracted fields are configured through a combination of props.conf, transforms.conf and fields.conf. Before manually modifying any configuration file, read about configuration files.

Previous: Train Splunk to recognize a timestamp | Next: Create indexed fields via configuration files

Comments

That's correct. I've updated this page to make that more clear.
Posted by emma on Mar 24 2008, 10:51am
Doesn't KV_MODE = none only disable automatic extraction, and not fields entirely, as the section claims? In particular, you will still get any fields explicitly defined in props+transforms, right?
Posted by goldburtd on Mar 21 2008, 5:41pm