Advanced conditional alerting
This documentation applies to the following versions of Splunk: 4.0 , 4.0.1 , 4.0.2 , 4.0.3 , 4.0.4 , 4.0.5 , 4.0.6
Advanced conditional alerting
If you base your alert conditions on a saved search, you can create specific conditions for triggering alerts and reduce the number of false positive alerts. Also, you can trigger alerts with more complex calculations on the result set without changing the content of the alert.
How to define alert conditions for saved searches
Note: This topic assumes that you already have a saved search.
To define alert conditions based on a saved search:
1. In savedsearches.conf define the conditions to trigger an alert.
In the stanza for your saved search, add the line:
alert_condition = <string>
The string is a search that evaluates on the results of the saved search. This condition triggers an alert action if the search yields a result.
2. In savedsearches.conf, specify the type of alert action to trigger and define the arguments associated with the action.
Within the same saved search stanza, specify the alert action you want to trigger when your conditions are met. Actions include sending email, enabling and publishing to RSS, and running a script; you can specify more than one action.
a. If you want send an email alert, add the line:
action_email = <list of email addresses>
This specifies a comma-delimited list of email addresses that will receive the alert.
b. If you want to trigger a shell script, add this line:
action_script = <script>
This specifies the name of the shell script to run when the alert conditions are met; script should be located in $SPLUNK_HOME/bin/scripts.
3. In alert_actions.conf, define your alert action.
a. For email alerts, you already specified the email addresses that will receive the alerts in savedsearches.conf. Here, you can define the sender, subject and format of the email and the SMTP mailserver to use when sending the email.
4. Restart Splunk to implement your changes to the configuration files.
Example of conditional email alert
Let's say you have a saved search called access_combined_error, that searches for:
sourcetype=access_combined error
You want to be alerted whenever this error occurs more than 5 times on any host machine.
To set this alert condition, first edit savedsearches.conf. Your stanza may look like this:
[access_combined_error] search = sourcetype=access_combined error role = Admin alert_condition = | stats count by host | where count>5 action_email = me@myhost.com
Next, edit alert_actions.conf to format your email alert. Your stanza may look like this:
[email] from = alert@mysplunk.com subject = access combined error format = plain
Finally, restart Splunk to implement your configuration changes.