• Overview
  • Overview
• User Interface
  • User Interface
    • Skinning SplunkWeb
  • Splunk toolbar for Firefox
  • Permalinks
    • Parameters
    • Unsupported Parameters
    • Examples
  • Configuring SplunkWeb
    • Skins
    • Localization
  • Scripting the Splunk search command
  • REST API
    • Messages
    • Authentication
    • Search
    • Enabling/disabling REST configuration options
    • Examples
  • SOAP API
    • How Splunk uses SOAP
    • Using the PCL to Make SOAP Requests
    • Additional PCL Search Examples
    • Creating SOAP Requests
    • SOAP Message Examples
    • SOAP error messages
• Data Inputs
  • Data Inputs
  • Input config
  • Scripted inputs
    • Configuration
    • Example
  • Understanding modules and processors
    • Default processors and pipelines
    • Pipeline data keys
    • Changing the default parsing and indexing sequence
    • Adding custom processors
  • Coding C/C++ processors
    • Writing Your Callback Function
    • Write Main
    • Add a Processor Config Stanza to Pipeline
  • Complete C Example
  • Complete C++ Example
  • Selecting events to process with a bundle
    • Create the Processor
    • Create the Module
    • Create the Bundle
    • Add an Additional Processor
• Data Outputs
  • Data Outputs
  • Customizing alert options
    • Email alerts
  • CLI for search
    • Actions
    • Default Argument
    • Parameters
    • Example
  • TCP Output
  • XML API for data output
• Management
  • Management
  • Authentication
    • Command Authentication
    • Parameters
    • Examples
    • Session Authentication
    • Examples
  • CLI for management
    • User Accounts
    • Controls
  • Configuring splunkd
    • Creating default logins on first startup
    • Changing the SSL configuration
  • XML API for management

Selecting events to process with a bundle

You can create a bundle to configure your module, just as you can use bundles with Splunk standard modules. Below is an example using a bundle to only send certain events, 404 messages from an apache log file, to the custom module.

Create the Processor

This example uses the sample C++ module described on the previous page, here referred to as "sampleProcessor".

Create the Module

Make a new directory under $SPLUNK_HOME/etc/modules. For this example, it is called "testModule" and the sampleProcessor executable file goes there.

Next, create a new config.xml with a new pipeline and the desired processors. This pipeline calls "readerin", "sampleProcessor" and "sendOut". readerin is configured to get data from the queue "testQueue", sampleProcessor appends the text "{xyzzy}" to the raw event and sendOut passes it to indexQueue to continue to the indexer.

Note You can also specify the full path instead of using "$$SPLUNK_HOME]]"

<module>
  <pipeline name="test-pipeline" type="startup">

    <processor name="readerIn" plugin="queueinputprocessor">
      <config>
        <queueName>testQueue</queueName>
      </config>
    </processor>

    <processor name="myProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>xyzzy</addToRaw>
      </config>
    </processor>

    <processor name="sendOut" plugin="queueoutputprocessor">
      <config>
        <queueName>indexQueue</queueName>
      </config>
    </processor>

  </pipeline>
</module>

Create the Bundle

Make a new directory under $SPLUNK_HOME/etc/bundles, here also called testModule. In it, create three new files:

Note Replace testbox.splunk.com with your hostname and the tail path with your desired path.

inputs.conf specifies the files we want to tail and assigns a sourcetype.

host = testbox.splunk.com

[tail:///var/log/httpd]
disabled = false
host = testbox.splunk.com
sourcetype = access_log

props.conf specifies that events with sourcetype access_log should use the regular expression configured in regexes.conf for test-pipeline. "TRANSFORMS-test" specifies the class ("-test") to identify a particular configuration stanza. Class names must be unique for each Splunk instance to avoid overriding existing behavior.

[access_log]
TRANSFORMS-test = test-pipeline

transforms.conf defines test-pipeline to look for the specified regular expression and send those events to a queue named "testQueue".

[test-pipeline]
REGEX = \s404\s
DEST_KEY = queue
FORMAT = testQueue

Note that the new 3.x format is transforms.conf instead of regexes.conf and TRANSFORMS-classname for REGEXES-classname.The complete bundle looks like this:

testbox [root]:/opt/splunk/etc/bundles$ ls testModule
inputs.conf     props.conf      transforms.conf

Restart splunk to load the new pipeline. The 404 events should now show the specified text appended:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy}

Add an Additional Processor

You can configure several processors in the same pipeline. For this example, to add a second instance of sampleProcessor, add the XML for a second processor with a different name. It can reference the same executable as myProcessor.

    <processor name="myOtherProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>You are in a maze of twisty little passages, all alike</addToRaw>
      </config>
    </processor>

Then, the text of the event would look like this:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy} {You are in a maze of twisty little passages, all alike}