• Overview
  • Overview
• User Interface
  • User Interface
    • Skinning SplunkWeb
  • Splunk toolbar for Firefox
  • Permalinks
    • Parameters
    • Unsupported Parameters
    • Examples
  • Configuring SplunkWeb
    • Skins
    • Localization
  • Scripting the Splunk search command
  • REST API
    • Messages
    • Authentication
    • Search
    • Enabling/disabling REST configuration options
    • Examples
  • SOAP API
    • How Splunk uses SOAP
    • Using the PCL to Make SOAP Requests
    • Additional PCL Search Examples
    • Creating SOAP Requests
    • SOAP Message Examples
    • SOAP error messages
• Data Inputs
  • Data Inputs
  • Input config
  • Scripted inputs
    • Configuration
    • Example
  • Understanding modules and processors
    • Default processors and pipelines
    • Pipeline data keys
    • Changing the default parsing and indexing sequence
    • Adding custom processors
  • Coding C/C++ processors
    • Writing Your Callback Function
    • Write Main
    • Add a Processor Config Stanza to Pipeline
  • Complete C Example
  • Complete C++ Example
  • Selecting events to process with a bundle
    • Create the Processor
    • Create the Module
    • Create the Bundle
    • Add an Additional Processor
• Data Outputs
  • Data Outputs
  • Customizing alert options
    • Email alerts
  • CLI for search
    • Actions
    • Default Argument
    • Parameters
    • Example
  • TCP Output
  • XML API for data output
• Management
  • Management
  • Authentication
    • Command Authentication
    • Parameters
    • Examples
    • Session Authentication
    • Examples
  • CLI for management
    • User Accounts
    • Controls
  • Configuring splunkd
    • Creating default logins on first startup
    • Changing the SSL configuration
  • XML API for management

Coding C/C++ processors

Processors allow for custom actions that cannot be done by configuration alone. But note that many things that required a processor in previous versions are now supported without custom code.

Developing C/C++ processors for the Splunk server follows a simple development model where a user supplies a function/method that gets called by splunkd for every event processed. A data structure is passed to this function that contains the event data and meta data. Helper functions are provided to manipulate portions of the data structure such as setting host, adding meta data, etc.

There are 3 things that need to be done to build and hook up your own processor:

1. Write a callback function
2. Write a program with a main to register your function
3. Add a config stanza that specifies your external processor

Below is a brief overview of each of the three steps. More complete examples can be found on on the Processor C Example and Processor C++ Example pages.

Writing Your Callback Function

The first thing to do is to implement a function with the following signature, this function will be called once for each event during processing. The actual name of the function need not be processEvent, it can be called anything you like.

     static int processEvent( struct SplunkTransaction *trans);

You will need to #include the header file splunk-extplugin.h located in SPLUNK_HOME/include.
An online version is on the splunk-extplugin.h page.

This header file defines helper functions that can be used to easily and safely manipulate the event data. There are several types of helper functions:

• Get/Set Functions
• Logging Functions
• Configuration Functions
• Error & Process management Functions

Get/Set Functions

These are used for manipulating the event and event meta data such as Host, Source, Sourcetype, the raw data, etc. Below are some of these functions, more can be found in the header file.

extern const char *SplunkGetSource(struct SplunkTransaction *trans);
extern void SplunkSetSource(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetSourceType(struct SplunkTransaction *trans);
extern void SplunkSetSourceType(struct SplunkTransaction *trans, const char *nval);

Logging Functions

The following functions provide logging facilities to the Splunk logs located in SPLUNK_HOME/var/log/splunk/splunkd.log.

extern void SplunkDebug(const char *fmt, ...)
extern void SplunkInfo(const char *fmt, ...)
extern void SplunkWarn(const char *fmt, ...)
extern void SplunkError(const char *fmt, ...)
extern void SplunkFatal(const char *fmt, ...)

Start splunk with the --debug option to see messages generated with the SplunkDebug function.

Configuration Functions

The following function provides key/value access to the tags in the external processes xml config (see below.)

extern const char *SplunkInstanceConfig(const char *key);

Error and Process Management

The following function provides a clean way to exit your custom processor.

extern void SplunkExit(int rval, const char *fmt, ...)

Write Main

The external processor is a separate application that is invoked during the Splunk startup process. The main() function is called, allowing for initialization and the hooking up of the event processing function described above.

The following example is the main function for the processor that just hooks up the event processing function. The function SplunkProcess_SingleThreaded registers your function so splunkd knows how to call it.

int main(int argn, char * const argv[])
{
    SplunkProcess_SingleThreaded(processEvent);
}

Note If you run your program from the command line, you will see the message "I/O Error -- error writing: Socket operation on non-socket." This is normal.

Add a Processor Config Stanza to Pipeline

To get your processor called, you must add it to a pipeline. Create a new module and pipeline for your processer and insert the XML for your processor in your module's config.xml.

The following XML stanza defines the external processor:

NOTE Replace the path in <command> ... </command> with the actual path to your executable, it can be anywhere but we recommend putting it in your custom module's directory. If your program requires command-line arguments, you can include them here although a better way to handle this is through config.xml. (See the complete examples for more on this.) extcmdprocessor is the built-in processor that handles custom processors.

This example inserts sampleProcessor into the main indexing pipeline. This means it will execute for every event Splunk indexes, including internal ones. If you want to only process some events, create a new pipeline as shown in the Using a Bundle with a Module example.

<processor name="extcmdprocessor" plugin="extcmdprocessor" action="insertAfter" target="indexIn" pipelineTarget="indexerPipe" >
      <config>
       <command>$$SPLUNK_HOME]]/etc/modules/myModule/sampleProcessor</command>  <!--  $$SPLUNK_HOME]] references your splunk installation directory -->
      </config>
     </processor>

Restart splunk to load your module and register your processor with splunkd. Errors, warnings and your log messages can be found in the splunkd.log file. Note that if you have output any messages using SplunkDebug(), you must run Splunk with "splunk start --debug" to have them generated. In the splunkd.log file you will also see messages like this indicating your module has been loaded:

02-20-2007 09:27:41.846 INFO loader - Processing Module ----> /opt/splunk/etc/modules/myModule
02-20-2007 09:27:41.851 INFO loader - Overlaying processors
02-20-2007 09:27:41.851 INFO loader - Overlaying processor extcmdprocessor into pipeline. Target=indexIn Action=insertAfter
02-20-2007 09:27:41.851 INFO loader - processor extcmdprocessor overlay completed
02-20-2007 09:27:41.854 INFO splunklogger - Module /opt/splunk/etc/modules/myModule loaded