Set up Alerts

Alerts in Splunk are equivalent to cron or batch jobs. An alert is comprised of: saved search terms, a schedule to run it, rules to trigger it and actions to perform. Alerts can be sent via email or RSS, or used to trigger a shell script. Any saved search can be turned into an alert.

via SplunkWeb

You can set up alerts either through the Saved Search tab of the Admin link. By clicking on any saved search, you can set up an alert:

Or you can configure an alert through the saved search link in the drop down menu next to the search box. Click on the Schedule & Alerts tab:

Alert Options

Schedule

Fill in the blanks to run the Alert at regular intervals. Too many searches running every minute or less can slow down the server.
- Click Advanced scheduling to specify cron-style scheduling.
- Please note: Splunk supports most standard cron notation. For example:
  - enter */60 * * * 1-5 to run your search every minute, Monday through Friday.

Alert Rules

Fill in the blanks to configure the Alert to trigger. The final value is a digit, for example Create an alert if the number of events is greater than 3.

Alert Actions

Splunk can send email, create an RSS feed, or run a shell command when an alert triggers. There are multiple variables you can pass to an email or shell script.

Alert History

This page shows what alerts have been triggered on the server since it was last initialized. To access, click the Admin link in the upper right hand corner and select the Saved Searches tab. Your alerts will show up in the Alert History column.
- Please note: If you need to send email alerts from a different host than your Splunk Server, see this section of the Developer Guide.

Specify which fields to show

When you receive alerts, any fields included in your search will also be displayed. You can edit the saved search to change which fields are displayed in your alert.

To eliminate a field, pipe your search to fields - $FIELDNAME. To add a field, pipe your search to fields + $FIELDNAME. You can add or subtract any number of fields -- just separate them with a comma: fields - $FIELD1, $FIELD2 + $FIELD3, $FIELD4.

For example:

GenericJDBCException starthoursago::01 | fields - sourcetype

This search will keep the sourcetype field from appearing in your alerts.

via configuration files

You can configure the above aspects through savedsearches.conf. Copy savedsearches.conf from $SPLUNK_HOME/etc/bundles/default/ into $SPLUNK_HOME/etc/bundles/local/. Open $SPLUNK_HOME/etc/bundles/local/savedsearches.conf file. You can edit all of the fields below:

[<Splunk name>]
        * Name of the saved Splunk (header)

query = <string>
        * Actual query terms of this saved Splunk (eg. index::sampledata http NOT 500)

isglobal = <integer>
        * If isglobal is set to 1, everyone can see/use this Splunk
        Possible values: 1/0.

counttype = <string>
    * Which count to compare the quantity against relation
    Possible values: "number of events", "number of hosts",
        "number of sources", "number of sourcetypes"

relation = <string>
    * How to compare against count
    Possible values: "greater than", "less than", "equal to",
        "drops by", "rises by"

quantity = <integer>
    * Number to compare against the number returned for the given counttype

schedule = <string>
    * Cron style schedule (i.e. */12 * * * *)

action_script = <string>
    * Name of shell script to run

action_rss = <integer>
    * Whether or not to create an rss link
    Possible values: 1/0

action_email = <string>
    * Comma delimited list of email addresses to send to

sendresults = <integer>
    * Whether or not to send the results along with the email/shell script
    Possible values: 1/0

For example:

[sudoalert]
action_rss = 1
counttype = number of events
disabled = false
enableSched = 1
isGlobal = 0
quantity = 0
query = sudo
relation = greater than
schedule = */12 * * * *
sendresults = 0
userid = 1

Script options

Your alert can trigger a shell script.

Specify the name of the script file to run when a saved search triggers an alert. You don't need to provide the path to the script if the script is located under $SPLUNK_HOME/bin/scripts/.

Command line arguments passed to the script are:

$1 - number of events returned when search run
$2 - searched terms saved at the time saved search was created
$3 - fully qualified query string, the way query is run internally
$4 - saved search name
$5 - trigger reason for the alert
$6 - encoded http link to the saved search results
$7 - tags that have been saved against this saved search, if there are any.

For examples on how scripts can be configured to work with alerts, please see send SNMP traps and send syslog events.

Previous: Set up saved searches | Next: Send SNMP traps

Comments

If you want fairly quick alerts when something new happens:
- create the search
- schedule the search, say for 5 minutes
- limit the search to "startminutesago=5"
- set the "If" "number of events" "greater than" "0"

As one might imagine, overlapping the schedule and startminutes will get you more alerts than you want. Setting "if" to "always" will alert even when no new events occur.
Posted by mgh4 on Dec 12 2007, 1:34pm