Create extracted fields via configuration files

Splunk automatically extracts fields during searches using known keywords for the source type and name/value pairs in the events. Examine the extracted fields in Splunk Web by clicking the Fields... link above the event display:

Also add additional extracted fields. Use these instructions to configure custom extracted fields.

Configuration

To create additional extracted fields, edit transforms.conf and props.conf.

transforms.conf

Add the following lines to $SPLUNK_HOME/etc/bundles/local/transforms.conf:

[$UNIQUE_STANZA_NAME]
REGEX = $YOUR_REGEX
FORMAT = $YOUR_CUSTOM_FIELD_NAME::$1

$UNIQUE_STANZA_NAME = name your stanza. Use this name later in configuring props.conf.
REGEX = create a regex that recognizes your custom field value.
FORMAT = $YOUR_CUSTOM_FIELD_NAME is the name of your extracted field; $1 is the value specified by the regular expression.
- In order to properly display field values containing whitespace in Splunk Web, you must apply quotes to the FORMAT key.
  - FORMAT = $YOUR_CUSTOM_FIELD_NAME::"$1"

Note: In order to preserve previous matching extractions, include a $0 in the FORMAT key. If you don't include $0, the previously extracted fields will be erased and only the last matching extraction specified in transforms.conf will be kept.

Note: Unlike configuring search fields, extracted field transforms.conf requires no DEST_KEY since nothing is being written to the index. The field is extracted at search time and is not persisted in the index as a key.

props.conf

Add the following lines to $SPLUNK_HOME/etc/bundles/local/props.conf:

[<spec>]
REPORT-$VALUE = $UNIQUE_STANZA_NAME

<spec> can be:
1. <sourcetype>, the sourcetype of an event.
2. host::<host>, where <host> is the host for an event.
3. eventtype::<eventtype> where <eventtype> is any valid event type that is either pre-defined in
Splunk or defined in eventtypes.conf.
NOTE: eventtype can only be used as a spec for creating extracted fields with REPORT<class>.
4. source::<source>, where <source> is the source for an event.

$UNIQUE_STANZA_NAME is the name of your stanza from props.conf.
$VALUE is any value you want to give to your stanza to identify its name-space.

To display only your explicitly configured extracted fields and not the automatically recognized ones, add KV_MODE = none to your stanza in transforms.conf.

Note: Extracted fields props.conf uses REPORT-$VALUE as opposed to TRANSFORMS-$VALUE used in configuring search fields.

Example

In this example, there is an error field that we wish to extract. The field can be identified by the occurrence of device_id= followed by a word within brackets and a text string terminating with a colon. The source type of the events is testlog.

In transforms.conf add:

[netscreen-error]
REGEX = device_id=[^ ]+\s+\[w+\](.*)(?
FORMAT = err_code::$1

In props.conf add:

[testlog]
REPORT-netscreen = netscreen-error

Extract fields from multi-line events

To extract fields from multi-line events, you must enable the multi-line of Splunk's regular expression processor. Turn on multi-line mode by including (?m) at the beginning of a regular expression.

When the regular expression processor is in multi-line mode( (?m) at the start of a regex pattern), the ^ and $ characters denote the beginning and ending of lines instead of the beginning and ending of the entire string.

Example

Below is an example of a log event from an application log. When a support engineer views this log, the customer wants to protect some of the information. For example SessionId and Ticket. They would like to mask these ID's except the last 4 characters. Ex. SessionId=###########7BEA&Ticket=############96EE

"2006-09-21, 02:57:11.58", 122, 11, "Path=/LoginUser Query=CrmId=ClientABC&ContentItemId=TotalAccess&SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&SessionTime=25368&ReturnUrl=http://www.clientabc.com, Method=GET, IP=209.51.249.195, Content=", ""
"2006-09-21, 02:57:11.60", 122, 15, "UserData:<User CrmId="clientabc" UserId="p12345678"><EntitlementList></EntitlementList></User>", ""
"2006-09-21, 02:57:11.60", 122, 15, "New Cookie: SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&CrmId=clientabc&UserId=p12345678&AccountId=&AgentHost=man&AgentId=man, MANUser: Version=1&Name=&Debit=&Credit=&AccessTime=&BillDay=&Status=&Language=&Country=&Email=&EmailNotify=&Pin=&PinPayment=&PinAmount=&PinPG=&PinPGRate=&PinMenu=&", ""

To anonymize the data you will need to modify your props.conf and transforms.conf files in the $SPLUNK_HOME/etc/bundles/local directory. In your props.conf add the following:

    [source::source-to-anonymize]
    TRANSFORMS-anonymize = session-anonymizer, ticket-anonymizer

In your transforms.conf you will want to add:

    [session-anonymizer]
    REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
    FORMAT = $1SessionId=########$2
    DEST_KEY = _raw

    [ticket-anonymizer]
    REGEX = (?m)^(.*)Ticket=\w+(\w{4}&.*)$
    FORMAT = $1Ticket=########$2
    DEST_KEY = _raw

Previous: Create indexed fields via configuration files | Next: Create extracted fields via Splunk Web

Comments

You have to restart Splunk in order to reload the configurations. Props.conf affects indexing properties and is thus loaded only upon restart of splunkd.
Posted by emma on Apr 15 2008, 5:15pm
How can you have Splunk re-read the transforms.conf file to look for new/changed extracted fields without restarting Splunk? Splunkweb obviously does this. It would be nice to have this information documented on this page so we aren't constantly restarting Splunk.
Posted by meeas on Mar 08 2008, 3:45pm