Alert with Splunk

Intro

Onscreen illustration:
Show a Splunk server icon in the middle with data flowing in from the stack of components as in Index IT but add lines leading to a schedule icon which in turn has lines leading to an email message icon, RSS feed icon and script icon.

Onscreen bullets:

* Save and schedule any search as an alert
o Alert_vo_00: "Now that you've indexed and searched some data, I'll show you how to run any search on a schedule and trigger alerts and actions via email, RSS, SNMP or scripts. It's a great way to proactively find problems and monitor user or system activities across lots of different technologies."

* (launch a browser window in the background to gmail, user:splunkalerts@gmail.com, pass:harharhar! YOU MUST BE USING FIREFOX FOR THIS DEMO TO WORK.)

Screencast

* (Use this search: 200 sourcetype::access_combined | where status=200 | where [search sourcetype::access_combined | where status<>200 | top file | where count>2 | fields + file | format] )

Alert_vo_01: "Let's say a customer on your website reports a transaction failing. One of the searches you did to investigate the problem will make a great alert. It finds all the web failures that slipped through the cracks of your other monitoring systems. Let's go run it on a schedule and get Splunk to alert you when this kind of transaction fails again."

* (Choose save search from menu)
* (Type the name "Files with intermittent errors")
* (Check Yes to share with all users.)

Alert_vo_02_B: "Click on the search menu and choose save. Type in the name you want to give the search. You can also share the search with other users. Its a great way to spread knowledge across your team."

* (Click on Schedule and Alerts)
* (Check run this search on a schedule and pick every minute)

Alert_vo_03: "Now set the schedule on which you'd like to run this search. Choose from predefined schedules or custom schedules."

* (Under Alerts, set the 'If' property - if # of events...greater than...0)

Alert_vo_04: "Alerts can trigger based on a variety of conditions, thresholds and changes including the number of events, hosts, or sources in your results. You set this one to trigger if the number of events is greater than 0."

* (Tick the 'Create an RSS feed' box)
* (Send email to "splunkalerts@gmail.com")
* (Choose include results)
* (Click save)

Alert_vo_05: "Now select the way you want to be notified when this alert triggers. You can optionally choose to have the results included with the alert."

* (Note for all of the below: because of bugs where alerts don't fire with report searches, we'll have to capture the screenshots/movies of results based on a simplified search for just 'sourcetype::access_common 200 | where status=200'. Josh already has this for the email one.)
* (Show Gmail email alert: Change the screen to my gmail account and show the alerts with results included.)
* (Show RSS reader with alert: Mark B is working on a screenshot of this. Show RSS reader with merge of feeds from multiple Splunk searches of which one is titled "Files with intermittent errors.")

Alert_vo_06: "You can watch your alerts from your email. Or from your RSS feed reader"

* (Show Nagios console with Splunk alert: Mark C is providing a Nagios console where we can see alerts called "Files with intermittent errors" as well as other Splunk and non-splunk alerts. Take a movie of this screen and click on link in the alert back to Splunk.)

* (Click on an Alert to launch another browser window with the search that generated the alert. This will not launch the correct search. Replace with a screenshot of the original search.)

Alert_vo_08: "What I really like is how you can trigger scripts that send alerts and events to other applications like monitoring systems using SNMP. The alerts can be displayed on my monitoring console along with a link to launch the original search."

* (Do a search for "sourcetype::ps NOT weblogic" last 60 minutes, open save search dialog, switch to schedule and set to run every minute, and enter the following into the "Trigger shell script" field: "/opt/bea92/user_projects/domains/petstore/bin/startWebLogic.sh" - take a screenshot of this and swap out with a nice transition to a screenshot of a terminal with the actual script in it. I will send the script screenshot.)

Alert_vo_08.1: "Here's a great example of an alert that restarts an application server when Splunk notices it isn't running. It triggers a script to take an automated action."

Onscreen illustration:
Show a Splunk server icon in the middle with data flowing in from the stack of components as in Index IT but add lines leading to a schedule icon which in turn has lines leading to an email message icon, RSS feed icon and script icon.

Alert_vo_09: "With just a few clicks Splunk can take any search and turn it into a proactive alert. You can improve your monitoring across multiple systems and technologies by alerting on all your IT data. Now you can find your problems before someone else does!"