This documentation applies to the following versions of Splunk: 4.0 , 4.0.1 , 4.0.2 , 4.0.3 , 4.0.4 , 4.0.5 , 4.0.6
After you schedule a search, you can configure an alert based on that search.
You can set up a basic conditional alert, which involves thresholds in the number of events, sources, or hosts in your results. If these conditions are met, Splunk notifies you via email, triggering the execution of a custom script or RSS feed.
Alternatively, you can define an advanced conditional alert which is based on a search string evaluated on the results of the scheduled search. This topic explains how you can set up both kinds of searches.
After you define the alert, you can set a notification method (email/script/rss) that Splunk can use to tell you when the alert is triggered.
Follow this procedure to define a basic conditional alert that notifies you of thresholds related to events, hosts, or sources.
1. In the Perform action dropdown menu, choose either if number of events, if number or sources, or if number of hosts. Choosing one of these three values causes the next two fields for basic conditional alerting to appear. (Alternatively, you can choose Always to have Splunk notifiy you each time the search is run. This can be handy if the search runs on an infrequent basis and you just want to see the results no matter what they are.)
2. Choose a comparison operation from the dropdown list that appears below the Perform actions field: is greater than, is less than, is equal to, rises by, or drops by.
3. In the field adjacent to the comparison operation dropdown, enter an integer for the alert threshold.
For example, you can set up an alert that notifies you if number of events returned by the search is greater than a threshold of 10.
Advanced conditional alerting enables you to define alert conditions based on the results of a conditional search that is applied to the results returned by the scheduled search. Splunk triggers the alert if the conditional search returns at least one result.
By basing your alert conditions on a search, you can define specific conditions for triggering alerts and reduce the incidence of false positive alerts.
Follow this procedure to define an advanced conditional alert:
1. In the Perform action dropdown menu, select if custom condition is met. The Custom condition search field appears.
2. Enter your conditional search in the Custom condition search field.
For example, you could set up an advanced conditional alert that notifies you if the conditional search returns more than 100 results using the following Custom condition search: stats count | search count > 100
Define how you want Splunk to notify you. You can have Splunk notify you via email. You can also have Splunk set off a shell script that runs a process when the alert is triggered.
To have Splunk contact you and other stakeholders via email, enter the relevant email addresses in the Send email field, separated by commas.
To have Splunk send alert emails, you first need to configure the email alert settings. You can do this in Splunk Web or by editing the alert_actions.conf. In Splunk Web, navigate to Manager > Email alert settings. Here you can define the Email format and specify whether or not you want Splunk to Include results inline.
Note: If you don't see Email alert settings, you do not have permission to edit the settings. In this case, contact your Splunk Admin.
When you schedule the alert, select Include results in email in the Save search dialogue box if you want Splunk to include the alerting search results in the alert email.
If you want Splunk to trigger a shell script when the alert conditions are met, select Trigger a shell script and enter the file name of the script that you want Splunk to execute. For example, you may want an alert to trigger a script that generates an SNMP trap notification and sends it to another system such as a Network Systems Management console. Meanwhile, you could have a different alert that--when its alert conditions are met--triggers a script that calls an API, which in turn sends the triggering event to another system.
Note: For security reasons, all alert scripts must be placed in the $SPLUNK_HOME/bin/scripts directory. This is where Splunk will look for any script triggered by an alert.
For more details on configuring alerts, including instructions for configuring alerts using savedsearches.conf, see the Admin Manual topic on alerts.
When you receive alerts, Splunk includes all the fields in your search. Edit the saved search to specify which fields you want included and excluded.
fields - $FIELDNAME.
fields + $FIELDNAME.
You can specify multiple fields to include and exclude in one string. For example, your Search field may be:
yoursearch | fields - $FIELD1,$FIELD2 + $FIELD3,$FIELD4
The alert you receive will exclude $FIELD1 and $FIELD2, but include $FIELD3 and $FIELD4.
Summary indexing is an action that you can configure for any scheduled search. You use summary indexing when you need to perform analysis/reports on large amounts of data over long timespans, which typically can be quite time consuming, and a drain on performance if several users are running similar searches on a regular basis.
With summary indexing, you define a scheduled search that computes sufficient statistics (a summary) for events covering a time slice. Each time Splunk runs the search it saves the results into a summary index that you've designated. You can then search and report on this smaller (and thus faster) summary index instead of working with the much larger dataset that the summary index is based on.
If you want to enable summary indexing for this search, select Enable summary indexing. For more information about summary indexing and setting up summary indexing through Splunk Web, see "Use summary indexing for increased reporting efficiency" in this manual.