This documentation does not apply to the most recent version of Splunk.
This documentation applies to the following versions of Splunk: 2.1 , 2.2 , 2.2.1 , 2.2.3 , 2.2.6
Splunk-2-Netcool is an integrated module that provides seamless workflow and data integration between Splunk Professional and IBM Tivoli Netcool. It allows Netcool customers to launch Splunk directly from the Netcool/Webtop and Netcool/OMNIbus Event List. It also configures Splunk to seamlessly index events from any Netcool ObjectServer, to provide the ability to search Netcool events alongside other kinds of IT data, such as logs and configuration files from servers and applications. Finally, it allows Live Splunk alerts to be forwarded to a Netcool ObjectServer for notification and correlation.
In addition for the included sample scripts you will need Perl as well as the Net-SNMP package installed.
The Splunk-2-Netcool module can be installed on systems where Splunk and Netcool are running on the same or on different servers. There are two tarballs included with this module, one each for installation into your Splunk home directory and into your Netcool home directory.
If you have any questions or need help with installation please contact Splunk Support.
Download and install the tarball splunk-2-netcool-v1.0-SPLUNK.tgz into your Splunk home directory ``$SPLUNK_HOME`` (the default is /opt/splunk) :
Download and install the tarball splunk-2-netcool-v1.0-NETCOOL.tgz into your Netcool home directory $NCHOME (the default is /opt/netcool):
Live Splunks are bookmarked searches scheduled to run automatically at regular intervals, just like a cron job. You can set up a Live Splunk to send an SNMP trap alert to your Netcool/Webtop consol. In this section we will configure an example Live Splunk to send an alert to your Webtop console whenever authentication errors are found in your syslog.
In order to handle the SNMP traps sent by the Live Splunk script you can use the file mttrapd.rules.splunk that was installed in $NCHOME/omnibus/probes/linux286/. Either cut and paste the contents into your main mttrapd.rules file, or use an include statement, and make changes as appropriate to work with your local environment. It should look something like the following:
if (nmatch($enterprise, ".1.3.6.1.4.1.27389.1")) ### Trap is a Live Splunk
{
@Manager = "MTTrapd Probe"
@Agent = "mttrapd"
@Class = "300"
@AlertGroup = "Live Splunk"
@Severity = 2
@Type = 0
@AlertKey = $4 # Live Splunk name
@Node = $6
@NodeAlias = $PeerIPaddress
@Summary = $7
update(@URL, TRUE)
@URL = $3
@Identifier = "" + @Node + "" + @Agent + "" + @AlertKey + "" + @Summary
}
Finally force the MT Trapd probe to re-read the main rules file by issuing the command kill -HUP pid on the MT Trapd process ID.
Follow the instruction here to Send an SNMP Trap from a Live Splunk. You will need to customize the script to send the trap to the Host:Port that you have installed the MT Trapd probe. You can find the Port by looking in the configuration file ``$NC_HOME/omnibus/probes/linux2x86/mttrapd.props``.
See the section below Using Splunk-2-Netcool to see a Live Splunk in Netcool.
Create a new Tool in Netcool/Webtop by logging in as an Administrator and going to the Webtop Admin screen and then clicking on the Tools link on the left:
Click the Create Tool button to get the Tool Editor dialog:
Name the tool SplunkIT, and leave the radio button on Execute CGI/URL, then click OK button to get to the second Tool Editor dialog:
For the URL enter the following, replacing the string YOURSPLUNKHOST with the host:port of your local Splunk installation:
``$(SERVER)cgi-bin/splunkit.cgi?SplunkHost=YOURSPLUNKHOST&TimeRangeSeconds=60&FieldsNetcoolToSplunk=LastOccurrence:starttime,Node:host``
This URL will enable you to Splunk for all log events across your infrastructure indexed by Splunk from the same host within a 60 second time window of the selected event. For more information on customizing the parameters passed to Splunk please see the file ``$NCHOME/etc/webtop/cgi-bin/splunkit.cgi``.
Click the ``Groups…`` button and set this tool to be accessible as necessary in your local environment, then click OK button:
Click the ``Fields…`` button and make sure that the checkboxes next to LastOccurrence and Node are checked:
Click the OK button to exit the Fields... dialog and OK button again to exit the Tool Editor dialog.
You now have a SplunkIT tool. Just right-click on any event in a Netcool/Webtop AEL, select SplunkIT, and a browser window will bring up Splunk showing all log events indexed by Splunk related to the selected event. See the section below Using Splunk-2-Netcool.
There is a second script installed livesplunk.cgi that can also be configured as a Tool, and used when you want to launch Splunk with exactly the same search that triggered the Live Splunk. To configure just follow the same steps as above to Create Tool, with the following differences:
LiveSplunkIT
``$(SERVER)cgi-bin/livesplunk.cgi``
Fields... dialog and make sure the checkboxes URL and AlertGroup are selected
In addition to all other logs and IT data in your infrastructure you can also use Splunk to index and search in realtime all events received by Netcool/OMNIbus.
Splunk has already been configured to receive events when you installed the tarball earlier. By default it listens on TCP port 2665. If necessary you can change this to a different port number in the following file:
$SPLUNK_HOME/etc/bundles/netcool/inputs.conf
You must restart Splunk in order to start listening for Netcool events, either using the GUI admin or the following command line:
``# $SPLUNK_HOME/bin/splunk restart``
Note that it is important to complete this step before proceeding, as the Netcool/OMNIbus Socket Gateway will not start unless Splunk is running and receiving events on the configured port.
This document assumes you have already obtained from IBM and installed on your system the necessary software and license keys for Netcool/OMNIbus Socket Gateway, gateway-nco-g-socket. Please refer to the documentation included with the Socket Gateway as well as the IBM Tivoli Micromuse Support resources.
Included in the Splunk-2-Netcool tarball is the following file:
``$NCHOME/omnibus/etc/G_SOCKET.conf.splunk``
This file is meant as a reference, you can either copy or rename it to $NCHOME/omnibus/etc/G_SOCKET.conf, or modify G_SOCKET.conf if it already exists. Edit the file to make sure that the HOST and PORT settings are correct for your local Splunk installation, replacing if necessary 'localhost' with the host name or IP address of the server running Splunk and leave the port set to 2665 unless you changed it above.
`` HOST = 'localhost', # host gateway is writing to``
`` PORT = 2665, # port gateway is writing to on 'HOST'``
The file G_SOCKET.conf should now look something like the following:
You will then need to start or restart your gateway in order to start sending events to your Splunk Server. Please see the instructions included with your gateway or refer to the documentation at IBM Tivoli Micromuse Support resources.
Please note that if you now start or restart Splunk it will cause the Netcool/OMNIbus Socket Gateway to stop, and it must restarted in order to continue sending events to Splunk. Another way to handle this situation is to configure the Netcool/OMNIbus process control agent, nco_pa.
Another option is to use the File Writer Gateway instead of the Socket Gateway. In this case use the included file ``$NCHOME/omnibus/etc/G_FILE.conf.splunk``. Copy this file or edit your G_FILE.conf to write events to a file:
`` FILE = '/opt/netcool/omnibus/log/NCOMS_G_FILE.out', # where to put the file``
Your G_FILE.conf should now look something like the following:
Additionally you will need to modify the file $SPLUNK_HOME/etc/bundles/netcool/inputs.conf to tail from the file created by the File Writer Gateway as follows:
[tail:///opt/netcool/omnibus/log/NCOMS_G_FILE.out]
disabled = false
host = localhost
sourcetype = netcool
You must restart Splunk in order to start listening for Netcool events, either using the GUI admin or the following command line:
``# $SPLUNK_HOME/bin/splunk restart``
Also note that If you restart Splunk it will cause the Netcool/OMNIbus File Writer Gateway to stop, and it must restarted in order to continue sending events to Splunk. Another way to handle this situation is to configure the Netcool/OMNIbus process control agent, nco_pa.
When a Live Splunk is received in your Netcool/Webtop console it will look like the following, including the URL/Permalink to bring up the Live Splunk in your browser:
To launch Splunk from Netcool/Webtop, right-click on an event in a view that contains an Active Event List and select SplunkIT:
For a quick Splunk tutorial see the Splunk User Manual.
An up-to-date Splunk User Manual and Admin Manual are available on our online Documentation. Technical support forums, FAQs and email contacts are available at Splunk Support.
IBM Netcool documentation, help and support are available at IBM Tivoli Micromuse Support resources.
