This documentation does not apply to the most recent version of Splunk.
This documentation applies to the following versions of Splunk: 3.3 , 3.3.1 , 3.3.2 , 3.3.3 , 3.3.4 , 3.4 , 3.4.1 , 3.4.2 , 3.4.3 , 3.4.5 , 3.4.6 , 3.4.8 , 3.4.9 , 3.4.10 , 3.4.11 , 3.4.12 , 3.4.13
In a distributed Splunk setup, you can send specific data to different queues for further processing. This topic discusses how to filter your data and send it specifically to nullQueue, or Splunk's /dev/null directory.
To filter certain events out before your data is indexed, use the instructions below to send those events to nullQueue.
Read more about how to filter and route to an alternate index.
Important: When you choose to filter your data depends on your distributed setup. However, the filtering needs to occur on the Splunk instance that parses the data; this may be either the indexer or the forwarder instance.
To filter out specific events:
1. Identify an attribute of the event that can be used to separate it from others.
2. Create an entry in props.conf for the source, source type or host and specify a TRANSFORMS class and a TRANSFORMS name. The class name refers to a regular expression stanza you will place in transforms.conf.
3. Create an entry in transforms.conf with a regular expression that matches the identified attributes (from Step 1) and sets the DEST_KEY to queue and the FORMAT key to a specific queue (indexQueue, parsingQueue, nullQueue, etc).
Use the $SPLUNK_HOME/etc/system/README/props.conf.example and ../transforms.conf.example as examples, or create your own props.conf and transforms.conf. Make any changes in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work.
In $SPLUNK_HOME/etc/system/local/props.conf add the following stanza:
[<spec>] TRANSFORMS-$name=$UNIQUE_STANZA_NAME
<spec> can be:
$NAME is whatever unique identifier you want to give to your transform.
$UNIQUE_STANZA_NAME must match the stanza name of the transform you create in transforms.conf.
In $SPLUNK_HOME/etc/system/local/transforms.conf add the following stanza:
[$UNIQUE_STANZA_NAME] REGEX = $YOUR_CUSTOM_REGEX DEST_KEY = queue FORMAT = nullQueue
Name your stanza with $UNIQUE_STANZA_NAME to match the name you specified in props.conf. Add $YOUR_CUSTOM_REGEX based on the attribute you've identified; it should specify the key term that identifies the events you want to remove.
Leave DEST_KEY and FORMAT with the above values to send identified events to the nullQueue (delete them before indexing).
This example sends all sshd events from /var/log/messages to nullQueue.
In props.conf:
[source::/var/log/messages] TRANSFORMS-null= setnull
In transforms.conf:
[setnull] REGEX = \[sshd\] DEST_KEY = queue FORMAT = nullQueue
For those using WMI to capture events from Windows machines, the syntax is specific in props.conf on the source. This example will allow you to filter out two different event codes (592 or 593) using an "or" statement in regex.
In props.conf:
[wmi] TRANSFORMS-foo=wminull
In transforms.conf:
[wminull] REGEX=(?m)^EventCode=(592|593) DEST_KEY=queue FORMAT=nullQueue
This example is the reverse of the previous. The user wants to keep only sshd events from /var/log/messages; everything else goes to nullQueue. In this case, you need to define two transforms.
In props.conf:
[source::/var/log/messages] TRANSFORMS-set= setnull,setparsing
In transforms.conf
[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = \[sshd\] DEST_KEY = queue FORMAT = indexQueue