Understanding modules and processors

Understanding modules and processors

Default processors and pipelines

The universal pipeline, or the parsing pipeline, is where events are input, processed and output to the indexing pipeline.

Modules, pipelines and queues

Below is a list of the processors, in order executed, that make up the default universal pipeline. You can see this in the pipeline "parsingPipeline" in $SPLUNK_HOME/etc/myinstall/splunkd.xml file.

readerIn : Queue Input processor, data comes in here

utf8 : UTF8 processor

linebreaker : Line Breaking

aggregator : Line Merging/Date Extraction

regexreplacement : Regex Extraction

typing : Event Typing

clusterer : Meta Event Creation

sendOut : Queue Output, data sent to next pipeline

Pipeline data keys

Changing the default parsing and indexing sequence

The processors that make up what is called the Universal pipeline can be reconfigured or replaced by creating a new module with your custom processing pipeline.

The Splunk processor loading architecture supports the ability to insert a processor before, after or instead of another processor. If you are extending splunk by providing your own processor or wish to change the processing pipelines defined in splunkd.xml it is recommended that you define your processor in a new module and use the "insertBefore", "insertAfter", or "replace" attribute.

For example, lets suppose you wanted to add a processor "replaceProcessor" just before the "indexer" processor:

1. Create a new module directory in $SPLUNK_HOME/etc/modules/replaceProcessor

1. Create a config.xml file in the new module directory

1. Define a module configuration with a list of processors - note that for this case you do not use the <pipeline> ... </pipeline> tags because you are not defining a new pipeline but changing an existing one.

1. Define your processor(s) in the config.xml file and specify the action and target and target pipeline.

• • action is a value : "insertBefore", "insertAfter" or "replace"
  • target is the name of processor to insert before/after or replace
  • pipelineTarget is the pipeline where the inserted or replaced processor exists.

Here is the config.xml for an example replaceProcessor (uses an example urlencodeprocessor.) It will insert the "replaceProcessor" before the "indexer" processor in the "indexerPipe" pipeline:

<module>
   <processor name="replaceProcessor" plugin="urlencodeProcessor" action="insertBefore" target="indexer" pipelineTarget="indexerPipe" >
       <config></config>
    </processor>
</module>

Adding custom processors

Custom processors can provide data input and handling not available by other methods. You can add to the existing default processing or replace it by changing the configuration of Splunk pipelines.