Create extracted fields via configuration files

Create extracted fields via configuration files

Splunk automatically extracts fields during searches using known keywords for the source type and name/value pairs in the events. Examine the extracted fields in Splunk Web by clicking the Fields... link above the event display:

Also add additional extracted fields. Use these instructions to configure custom extracted fields.

Configuration

To create additional extracted fields, edit transforms.conf and props.conf.

transforms.conf

Add the following lines to $SPLUNK_HOME/etc/bundles/local/transforms.conf:

[$UNIQUE_STANZA_NAME]
REGEX = $YOUR_REGEX
FORMAT = $YOUR_CUSTOM_FIELD_NAME::$1

• $UNIQUE_STANZA_NAME = name your stanza. Use this name later in configuring props.conf.
• REGEX = create a regex that recognizes your custom field value.
• FORMAT = $YOUR_CUSTOM_FIELD_NAME is the name of your extracted field; $1 is the value specified by the regular expression.
  • In order to properly display field values containing whitespace in Splunk Web, you must apply quotes to the FORMAT key.
  • FORMAT = $YOUR_CUSTOM_FIELD_NAME::"$1"

Note: In order to preserve previous matching extractions, include a $0 in the FORMAT key. If you don't include $0, the previously extracted fields will be erased and only the last matching extraction specified in transforms.conf will be kept.

Note: Unlike configuring search fields, extracted field transforms.conf requires no DEST_KEY since nothing is being written to the index. The field is extracted at search time and is not persisted in the index as a key.

props.conf

Add the following lines to $SPLUNK_HOME/etc/bundles/local/props.conf:

[<spec>]
REPORT-$VALUE = $UNIQUE_STANZA_NAME

<spec> can be:

1. <sourcetype>, the sourcetype of an event.
2. host::<host>, where <host> is the host for an event.
3. eventtype::<eventtype> where <eventtype> is any valid event type that is either pre-defined in

Splunk or defined in eventtypes.conf.

NOTE: eventtype can only be used as a spec for creating extracted fields with REPORT<class>.

1. source::<source>, where <source> is the source for an event.

$UNIQUE_STANZA_NAME is the name of your stanza from props.conf.

$VALUE is any value you want to give to your stanza to identify its name-space.

To display only your explicitly configured extracted fields and not the automatically recognized ones, add KV_MODE = none to your stanza in transforms.conf.

Note: Extracted fields props.conf uses REPORT-$VALUE as opposed to TRANSFORMS-$VALUE used in configuring search fields.

Example

In this example, there is an error field that we wish to extract. The field can be identified by the occurrence of device_id= followed by a word within brackets and a text string terminating with a colon. The source type of the events is testlog.

In transforms.conf add:

[netscreen-error]
REGEX = device_id=[^ ]+\s+\[w+\](.*)(?
FORMAT = err_code::$1

In props.conf add:

[testlog]
REPORT-netscreen = netscreen-error

Extract fields from multi-line events

To extract fields from multi-line events, you must enable the multi-line of Splunk's regular expression processor. Turn on multi-line mode by including (?m) at the beginning of a regular expression.

When the regular expression processor is in multi-line mode( (?m) at the start of a regex pattern), the ^ and $ characters denote the beginning and ending of lines instead of the beginning and ending of the entire string.

Example

Below is an example of a log event from an application log. When a support engineer views this log, the customer wants to protect some of the information. For example SessionId and Ticket. They would like to mask these ID's except the last 4 characters. Ex. SessionId=###########7BEA&Ticket=############96EE

"2006-09-21, 02:57:11.58", 122, 11, "Path=/LoginUser Query=CrmId=ClientABC&ContentItemId=TotalAccess&SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&SessionTime=25368&ReturnUrl=http://www.clientabc.com, Method=GET, IP=209.51.249.195, Content=", ""

"2006-09-21, 02:57:11.60", 122, 15, "UserData:<User CrmId="clientabc" UserId="p12345678"><EntitlementList></EntitlementList></User>", ""

"2006-09-21, 02:57:11.60", 122, 15, "New Cookie: SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&CrmId=clientabc&UserId=p12345678&AccountId=&AgentHost=man&AgentId=man, MANUser: Version=1&Name=&Debit=&Credit=&AccessTime=&BillDay=&Status=&Language=&Country=&Email=&EmailNotify=&Pin=&PinPayment=&PinAmount=&PinPG=&PinPGRate=&PinMenu=&", ""

To anonymize the data you will need to modify your props.conf and transforms.conf files in the $SPLUNK_HOME/etc/bundles/local directory. In your props.conf add the following:

	[source::source-to-anonymize]
	TRANSFORMS-anonymize = session-anonymizer, ticket-anonymizer

In your transforms.conf you will want to add:

	[session-anonymizer]
	REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
	FORMAT = $1SessionId=########$2
	DEST_KEY = _raw
	[ticket-anonymizer]
	REGEX = (?m)^(.*)Ticket=\w+(\w{4}&.*)$
	FORMAT = $1Ticket=########$2
	DEST_KEY = _raw