Tale of Tinkering with Splunk IT Service Intelligence and Notables

As you probably already know, Splunk IT Service Intelligence (ITSI) gives users even deeper service insights of their machine data. Using Episode Review data, analysts are able to gain awareness of the current status and level of severity of episodes occurring in the underlying system. Episodes are groups of notable events identifying anomalous incidents detected by an ITSI multi-KPI alert, a correlation search or an anomaly detection algorithm.

In response to Splunk ITSI episodes, users can execute actions such as pinging a host, running a script or creating a ticket in an external ticketing system. But what about sending an email that includes the URL links to the underlying deep dives to contributing KPIs and services? Well, Splunk ITSI doesn’t come with that “off the shelf."

Luckily, Splunk provides a python-based notable event action SDK (or Splunk ITSI Event Management SDK in the newest deployments), giving developers a simplified common basis to start creating their own custom episode actions.

Motivated by a specific request, my team and I spent some time investigating and putting together that custom email alert action, and I'm sharing our journey here in the hopes that it benefits you during your own development.

Prerequisites

1. You need to have the Splunk SDK for Python installed
2. SSH and UI access to your Splunk instance
3. If not already done, you need to configure your Splunk instance to send email notifications

Configuration

1. Create at least one custom python script containing the logic of your action
2. Verify script file(s) permission and ownership granting read to everyone and write to root user only
3. Add a custom alert action user interface to let users configure alert action properties. Its definition is given by an HTML fragment file placed in a specific app directory location 
4. Create a stanza in alert_actions.conf to configure and register the custom alert action for event management
5. Add a stanza (keeping the same stanza naming) for the custom alert action you want to perform in notable_event_actions.conf, which is used to configure actions to take on groups in Episode Review
6. Good practice is adding a stanza in alert_actions.conf.spec (same naming for consistency reasons), describing attributes and possible values for configuring global saved search actions of alert_actions.conf
7. Restart Splunk service to reload changes

Usage

As soon as Splunk is up and running again, you are ready to verify the installation and start using your new custom action.

1. Login into your Splunk instance
2. Enter the app IT Service Intelligence among available ones
3. Select Notable Events Review from the navigation bar
4. Click on the event you want to execute your custom action on
5. Depending on the value you assigned to parameter label in your stanza of alert_actions.conf, you should see that listed among Actions. An example highlighting the correspondence among configuration file and UI is provided in figure below
6. Click on your equivalent action to Custom send email to open your custom action UI pop-up and configure the action by filling fields such as
   • To: Email recipient
   • Subject: Email subject
7. Press Done to proceed

Your action should now have been executed successfully. Congratulations!

Feedback and Questions

We're eager to hear your thoughts, so please feel free to reach out with any feedback or questions you have. We are actively accepting issues and Pull Requests on GitHub; community contributions are most welcome.

----------------------------------------------------
Thanks!
Erica Pescio