• Overview
  • Overview
• User Interface
  • User Interface
    • Skinning SplunkWeb
  • Splunk toolbar for Firefox
  • Permalinks
    • Parameters
    • Unsupported Parameters
    • Examples
  • Configuring SplunkWeb
    • Skins
    • Localization
  • Scripting the Splunk search command
  • REST API
    • Messages
    • Authentication
    • Search
    • Enabling/disabling REST configuration options
    • Examples
  • SOAP API
    • How Splunk uses SOAP
    • Using the PCL to Make SOAP Requests
    • Additional PCL Search Examples
    • Creating SOAP Requests
    • SOAP Message Examples
    • SOAP error messages
• Data Inputs
  • Data Inputs
  • Input config
  • Scripted inputs
    • Configuration
    • Example
  • Understanding modules and processors
    • Default processors and pipelines
    • Pipeline data keys
    • Changing the default parsing and indexing sequence
    • Adding custom processors
  • Coding C/C++ processors
    • Writing Your Callback Function
    • Write Main
    • Add a Processor Config Stanza to Pipeline
  • Complete C Example
  • Complete C++ Example
  • Selecting events to process with a bundle
    • Create the Processor
    • Create the Module
    • Create the Bundle
    • Add an Additional Processor
• Data Outputs
  • Data Outputs
  • Customizing alert options
    • Email alerts
  • CLI for search
    • Actions
    • Default Argument
    • Parameters
    • Example
  • TCP Output
  • XML API for data output
• Management
  • Management
  • Authentication
    • Command Authentication
    • Parameters
    • Examples
    • Session Authentication
    • Examples
  • CLI for management
    • User Accounts
    • Controls
  • Configuring splunkd
    • Creating default logins on first startup
    • Changing the SSL configuration
  • XML API for management

Understanding modules and processors

Default processors and pipelines

The universal pipeline, or the parsing pipeline, is where events are input, processed and output to the indexing pipeline.

Modules, pipelines and queues

Below is a list of the processors, in order executed, that make up the default universal pipeline. You can see this in the pipeline "parsingPipeline" in $SPLUNK_HOME/etc/myinstall/splunkd.xml file.

readerIn

Queue Input processor, data comes in here

utf8

UTF8 processor

linebreaker

Line Breaking

aggregator

Line Merging/Date Extraction

regexreplacement

Regex Extraction

typing

Event Typing

clusterer

Meta Event Creation

sendOut

Queue Output, data sent to next pipeline

Pipeline data keys

Changing the default parsing and indexing sequence

The processors that make up what is called the Universal pipeline can be reconfigured or replaced by creating a new module with your custom processing pipeline.

The Splunk processor loading architecture supports the ability to insert a processor before, after or instead of another processor. If you are extending splunk by providing your own processor or wish to change the processing pipelines defined in splunkd.xml it is recommended that you define your processor in a new module and use the "insertBefore", "insertAfter", or "replace" attribute.

For example, lets suppose you wanted to add a processor "replaceProcessor" just before the "indexer" processor:

1. Create a new module directory in $SPLUNK_HOME/etc/modules/replaceProcessor
2. Create a config.xml file in the new module directory
3. Define a module configuration with a list of processors - note that for this case you do not use the <pipeline> ... </pipeline> tags because you are not defining a new pipeline but changing an existing one.
4. Define your processor(s) in the config.xml file and specify the action and target and target pipeline.
   • action is a value : "insertBefore", "insertAfter" or "replace"
     • target is the name of processor to insert before/after or replace
       • pipelineTarget is the pipeline where the inserted or replaced processor exists.

Here is the config.xml for an example replaceProcessor (uses an example urlencodeprocessor.) It will insert the "replaceProcessor" before the "indexer" processor in the "indexerPipe" pipeline:

<module>
   <processor name="replaceProcessor" plugin="urlencodeProcessor" action="insertBefore" target="indexer" pipelineTarget="indexerPipe" >
       <config></config>
    </processor>
</module>

Adding custom processors

Custom processors can provide data input and handling not available by other methods. You can add to the existing default processing or replace it by changing the configuration of Splunk pipelines.