Create fields via configuration files

Create fields via configuration files

Splunk automatically extracts fields during searches using known keywords for the source type and name/value pairs in the events. Examine the fields in Splunk Web by clicking the Fields... link above the event display:

You can add your own custom fields. Use the instructions below to create fields. The basic steps are:

1. Determine a pattern to identify the field in the event.

2. Write a regular expression to extract the field from the event.

3. Add your regex to transforms.conf.

4. In props.conf, link your regex to the source, source type or host containing the events.

5. If your field value is a portion of a word, you must also add an entry to fields.conf. See the example "create a field from a subtoken" below.

Configuration

To create additional fields, edit transforms.conf and props.conf. Edit these files in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.

Note: DO NOT edit files in $SPLUNK_HOME/etc/system/default/.

transforms.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/transforms.conf:

[<unique_stanza_name>]
REGEX = <your_regex>
FORMAT = <your_custom_field_name>::$1

• <unique_stanza_name> = name your stanza. Use this name later in configuring props.conf.
• <your_regex> = create a regex that recognizes your custom field value.
• FORMAT = <your_custom_field_name>::$1 is the name of your field; $1 is the value specified by the regular expression.
  • In order to properly display field values containing whitespace in Splunk Web, you must apply quotes to the FORMAT key.
  • FORMAT = <your_custom_field_name>::"$1"

Note: In order to preserve previous matching extractions, include a $0 in the FORMAT key. If you don't include $0, the previously extracted fields will be erased and only the last matching extraction specified in transforms.conf will be kept.

Note: Unlike configuring indexed fields, transforms.conf requires no DEST_KEY since nothing is being written to the index. The field is extracted at search time and is not persisted in the index as a key.

props.conf

Add the following lines to $SPLUNK_HOME/etc/system/local/props.conf:

[<spec>]
REPORT-<value> = <unique_stanza_name>

<spec> can be:

1. <sourcetype>, the sourcetype of an event.
2. host::<host>, where <host> is the host for an event.
3. source::<source>, where <source> is the source for an event.

• <unique_stanza_name> is the name of your stanza from transforms.conf.
• <value> is any value you want to give to your stanza to identify its name-space.

To display only your explicitly configured extracted fields and not the automatically recognized ones, add KV_MODE = none to your stanza in transforms.conf.

Note: Extracted fields props.conf uses REPORT-$VALUE as opposed to TRANSFORMS-$VALUE used in configuring indexed fields.

Examples

Add a new field

This examples shows how to create a new "error" field. The field can be identified by the occurrence of device_id= followed by a word within brackets and a text string terminating with a colon. The sourcetype of the events is testlog.

In transforms.conf add:

[netscreen-error]
REGEX = device_id=[^ ]+\s+\[w+\](.*)(?
FORMAT = err_code::$1

In props.conf add:

[testlog]
REPORT-netscreen = netscreen-error

Extract fields from multi-line events

This example shows how to anonymize fields in multi-line events.

Below is a sample event from an application log:

"2006-09-21, 02:57:11.58", 122, 11, "Path=/LoginUser Query=CrmId=ClientABC&ContentItemId=TotalAccess&SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&SessionTime=25368&ReturnUrl=http://www.clientabc.com, Method=GET, IP=209.51.249.195, Content=", ""

"2006-09-21, 02:57:11.60", 122, 15, "UserData:<User CrmId="clientabc" UserId="p12345678"><EntitlementList></EntitlementList></User>", ""

"2006-09-21, 02:57:11.60", 122, 15, "New Cookie: SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&CrmId=clientabc&UserId=p12345678&AccountId=&AgentHost=man&AgentId=man, MANUser: Version=1&Name=&Debit=&Credit=&AccessTime=&BillDay=&Status=&Language=&Country=&Email=&EmailNotify=&Pin=&PinPayment=&PinAmount=&PinPG=&PinPGRate=&PinMenu=&", ""

The administrator wants to protect some of the information, specifically the fields SessionId and Ticket. This example masks these IDs except the last 4 characters, for example: SessionId=###########7BEA&Ticket=############96EE

To anonymize the data, modify props.conf and transforms.conf in $SPLUNK_HOME/etc/system/local/.

Add the following to props.conf:

[source::source-to-anonymize]
TRANSFORM-anonymize = session-anonymizer, ticket-anonymizer

Now, configure transforms.conf to recognize multi-line data. To extract fields from multi-line events, you must enable the multi-line mode of Splunk's regular expression processor. Turn on multi-line mode by including (?m) at the beginning of a regular expression.

Add the following to transforms.conf:

[session-anonymizer]
REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
FORMAT = $1SessionId=########$2
DEST_KEY = _raw
[ticket-anonymizer]
REGEX = (?m)^(.*)Ticket=\w+(\w{4}&.*)$
FORMAT = $1Ticket=########$2
DEST_KEY = _raw

When the regular expression processor is in multi-line mode( (?m) at the start of a regex pattern), the ^ and $ characters denote the beginning and ending of lines instead of the beginning and ending of the entire string.

Create a field from a subtoken

If your field value is a smaller part of a token, you must add an entry to fields.conf. For example, your field's value is "123" but it occurs as "foo123" in your event.

Configure props.conf and transforms.conf as explained above. Then, add an entry to fields.conf:

[<fieldname>]
INDEXED = False
INDEXED_VALUE = False

• Fill in <fieldname> with the name of your field.
  • For example, [url] if you've configured a field named "url."
• Set INDEXED and INDEXED_VALUE to false.
  • This tells Splunk that the value you're searching for is not a token in the index.

For more information on using fields.conf, see the page on "configuring fields.conf".