Splunk for Netcool

Splunk-2-Netcool

Splunk-2-Netcool is an integrated module that provides seamless workflow and data integration between Splunk Professional and IBM Tivoli Netcool. It allows Netcool customers to launch Splunk directly from the Netcool/Webtop and Netcool/OMNIbus Event List. It also configures Splunk to seamlessly index events from any Netcool ObjectServer, to provide the ability to search Netcool events alongside other kinds of IT data, such as logs and configuration files from servers and applications. Finally, it allows Live Splunk alerts to be forwarded to a Netcool ObjectServer for notification and correlation.

System Requirements

• Splunk 3.0+, installed on any supported platform.

• IBM Tivoli Netcool running on a supported Linux platform: Red Hat Enterprise Linux AS, ES, WS versions 3 and 4, and SUSE Linux Enterprise Server 9.2, with the following components installed and running:
  • Netcool/OMNIbus, version 7.1+
  • Netcool/Webtop, version 2.0+
  • Netcool/OMNIbus SNMP probe, probe-nco-p-mttrapd
  • Netcool/OMNIbus Socket Gateway, gateway-nco-g-socket

In addition for the included sample scripts you will need Perl as well as the Net-SNMP package installed.

Install Splunk-2-Netcool

You can install the Splunk-2-Netcool module on systems where Splunk and Netcool are running on the same or on different servers. There are two tarballs included with this module, one each for installation into your Splunk home directory and into your Netcool home directory.

If you have any questions or need help with installation, contact Splunk Support.

Install Splunk-2-Netcool into Splunk

Download and install the tarball into your Splunk home directory $SPLUNK_HOME (the default is /opt/splunk ):

Install Splunk-2-Netcool into Netcool

Download and install the tarball into your Netcool home directory $NCHOME (the default is /opt/netcool):

Receive Splunk alerts in Netcool/Webtop

Splunk alerts are saved searches scheduled to run automatically at regular intervals, just like a cron job. You can set up an alert to send an SNMP trap alert to your Netcool/Webtop console. In this section, we will configure an example alert to send an alert to your Webtop console whenever authentication errors are found in your syslog.

Configure Netcool to Receive Live Splunk SNMP Trap

In order to handle the SNMP traps sent by the alert script, you can use the file mttrapd.rules.splunk that was installed in $NCHOME/omnibus/probes/linux286/. Either cut and paste the contents into your main mttrapd.rules file, or use an include statement, and make changes as appropriate to work with your local environment. It should look something like the following:

if (nmatch($enterprise, ".1.3.6.1.4.1.27389.1")) ### Trap is a Live Splunk
{
        @Manager = "MTTrapd Probe"
        @Agent = "mttrapd"
        @Class = "300"

        @AlertGroup = "Live Splunk"
        @Severity = 2
        @Type = 0
        @AlertKey = $4          # Live Splunk name
        @Node = $6
        @NodeAlias = $PeerIPaddress
        @Summary    = $7
        update(@URL, TRUE)
        @URL = $3
        @Identifier = "" + @Node + "" + @Agent + "" + @AlertKey + "" + @Summary
}

Finally, force the MT Trapd probe to re-read the main rules file by issuing the command kill -HUP pid on the MT Trapd process ID.

Send an SNMP Trap from an alert

Follow the instructions to send an SNMP trap from an alert. Customize the script to send the trap to the Host:Port that you have installed the MT Trapd probe. You can find the Port by looking in the configuration file $NC_HOME/omnibus/probes/linux2x86/mttrapd.props.

See the section below Using Splunk-2-Netcool to see an alert in Netcool.

Create a SplunkIT Tool in Netcool/Webtop

Create a new Tool in Netcool/Webtop by logging in as an Administrator and going to the Webtop Admin screen and then clicking on the Tools link on the left:

Click the Create Tool button to get the Tool Editor dialog:

Name the tool SplunkIT, and leave the radio button on Execute CGI/URL, then click OK to get to the second Tool Editor dialog:

For the URL enter the following, replacing the string YOURSPLUNKHOST with the host:port of your local Splunk installation:

$(SERVER)cgi-bin/splunkit.cgi?SplunkHost=YOURSPLUNKHOST&TimeRangeSeconds=60&FieldsNetcoolToSplunk=LastOccurrence:starttime,Node:host

This URL lets you search for all log events across your infrastructure indexed by Splunk from the same host within a 60 second time window of the selected event. For more information on customizing the parameters passed to Splunk, see the file $NCHOME/etc/webtop/cgi-bin/splunkit.cgi.

Click the Groups…button and set this tool to be accessible as necessary in your local environment, then click OK:

Click the Fields… button and make sure that the checkboxes next to LastOccurrence' and Node are checked:

Click the OK button to exit the Fields... dialog and OK button again to exit the Tool Editor dialog.

You now have a SplunkIT tool. Just right-click on any event in a Netcool/Webtop AEL, select SplunkIT, and a browser window will bring up Splunk showing all log events indexed by Splunk related to the selected event. See the section below **Using Splunk-2-Netcool**.

Create a LiveSplunkIT Tool in Netcool/Webtop

There is a second script installed livesplunk.cgi that can also be configured as a Tool, and used when you want to launch Splunk with exactly the same search that triggered the alert. To configure, just follow the same steps as above to Create Tool, with the following differences:

• In the first Tool Editor dialog name the Tool LiveSplunkIT
• In the second Tool Editor dialog set the URL to $(SERVER)cgi-bin/livesplunk.cgi
• In the Fields... dialog and make sure the checkboxes URL and AlertGroup are selected

Configuration to Index and Search Netcool Events

In addition to all other logs and IT data in your infrastructure you can also use Splunk to index and search in realtime all events received by Netcool/OMNIbus.

Configure Splunk to Receive

Splunk was already configured to receive events when you installed the tarball earlier. By default, it listens on TCP port 2665. If necessary you can change this to a different port number in the following file:

$SPLUNK_HOME/etc/apps/netcool/default/inputs.conf

You must restart Splunk in order to start listening for Netcool events, either using the GUI admin or the following command line:

# $SPLUNK_HOME/bin/splunk restart

Important: You must complete this step before proceeding, as the Netcool/OMNIbus Socket Gateway will not start unless Splunk is running and receiving events on the configured port.

Configure Netcool/OMNIbus Socket Gateway to Send

This document assumes you have already obtained from IBM and installed on your system the necessary software and license keys for Netcool/OMNIbus Socket Gateway, gateway-nco-g-socket. Refer to the documentation included with the Socket Gateway as well as the IBM Tivoli Micromuse Support resources.

Included in the Splunk-2-Netcool tarball is the following file:

$NCHOME/omnibus/etc/G_SOCKET.conf.splunk

This file is meant as a reference; you can either copy or rename it to $NCHOME/omnibus/etc/G_SOCKET.conf, or modify G_SOCKET.conf if it already exists. Edit the file to make sure that the HOST and PORT settings are correct for your local Splunk installation, replacing if necessary localhost with the host name or IP address of the server running Splunk and leave the port set to <code>2665 unless you changed it above.

HOST = 'localhost', # host gateway is writing to PORT = 2665, # port gateway is writing to on 'HOST'

The file G_SOCKET.conf should now look something like the following:

You will then need to start or restart your gateway in order to start sending events to your Splunk server. See the instructions included with your gateway or refer to the documentation at IBM Tivoli Micromuse Support resources.

Note: If you now start or restart Splunk it will cause the Netcool/OMNIbus Socket Gateway to stop, and it must restarted in order to continue sending events to Splunk. Another way to handle this situation is to configure the Netcool/OMNIbus process control agent, nco_pa.

Alternate Configuration of Netcool/OMNIbus File Writer Gateway to Send

Another option is to use the File Writer Gateway instead of the Socket Gateway. In this case use the included file $NCHOME/omnibus/etc/G_FILE.conf.splunk. Copy this file or edit your G_FILE.conf to write events to a file:

FILE = '/opt/netcool/omnibus/log/NCOMS_G_FILE.out', # where to put the file

Your G_FILE.conf should now look something like the following:

Additionally, you must modify $SPLUNK_HOME/etc/apps/netcool/default/inputs.conf to tail from the file created by the File Writer Gateway as follows:

[tail:///opt/netcool/omnibus/log/NCOMS_G_FILE.out]
disabled = false
host = localhost
sourcetype = netcool

You must restart Splunk in order to start listening for Netcool events, either using the GUI admin or the following command line:

# $SPLUNK_HOME/bin/splunk restart

Note: If you restart Splunk it will cause the Netcool/OMNIbus File Writer Gateway to stop, and it must restarted in order to continue sending events to Splunk. Another way to handle this situation is to configure the Netcool/OMNIbus process control agent, nco_pa.

Using Splunk-2-Netcool

Splunk alert in Netcool/Webtop

When an alert from Splunk is received in your Netcool/Webtop console, it will look like the following, including the URL/Permalink to bring up the alert in your browser:

Launch Splunk from Netcool/Webtop

To launch Splunk from Netcool/Webtop, right-click on an event in a view that contains an Active Event List and select SplunkIT:

For a quick Splunk tutorial see the Splunk User Manual.

More documentation, help, and support

Technical support forums, FAQs and email contacts are available at Splunk Support.

IBM Netcool documentation, help and support are available at IBM Tivoli Micromuse Support resources.