Documentation: 3.3
|  Print Version Contents
This page last updated: 11/12/08 03:11pm

Set up alerts via Splunk Web

Use Splunk Web to set up alerts. Follow these steps:

1. Create a saved search.
2. Schedule the search.
3. Define alert conditions.
4. Configure alert actions.

You can set up an alert at the time you create a saved search, or you can enable an alert on any existing saved search you have permission to edit.

Note: You must have email enabled on your Splunk server for alerts to be sent out. Alternately, your Splunk server must be able to contact your email server. Configure email settings by customizing alerts.

Create a saved search

First, set up a saved search:

  • Enter your search terms into the search bar and choose Save search... from the drop-down menu to the left of the search bar.
  • Fill in the fields to save your search and then click the Schedule & Output link at the top of the Save Search pop up.

Schedule the search

Next, schedule your search. This means your search runs on the specified schedule. For example, Splunk runs your search every hour or at midnight. If your search meets the alert conditions, then Splunk alerts you.

  • Check the box run this search on a schedule.
  • Choose either basic or cron scheduling.

Note: Too many searches running every minute can slow down the server.

Time ranges in a search

To get all the results from a set window of time, you may include a specific time range in your search, for example hoursago=1. Especially in distributed setups, data may not reach the indexer exactly when it is generated. Thus, it is a good idea to run your searches with a few minutes of delay.

For example, you want all the results from an hour time window, such as 4 PM to 5 PM.

  • Add the terms startminutesago=90 and endminutesago=30 to your search.
  • Then, schedule your search to run on the half hour using cron notation.

This ensures that you get all the results from the specified time period.

Define alert conditions

Now define alert conditions. Alert conditions tell Splunk whether or not to send you an alert. Enter a threshold number of events, sources, or hosts in your results. If the alert conditions are met, Splunk notifies you via email or RSS feed or triggers a shell script.

1. In the first drop-down menu under Alert when choose:

  • always
    • Splunk will always send you alerts when your search runs.
    • If you choose this option, all other conditions are grayed out in the second drop-down menu.
  • number of events
    • Splunk sends alerts only if the number of events your search returns matches the rest of the alert conditions.
  • number of sources
    • Splunk sends alerts only if the number of sources your search returns matches the rest of the alert conditions.
  • number of hosts
    • Splunk sends alerts only if the number of hosts your search returns matches the rest of the alert conditions.

2. In the second drop-down menu under Alert when choose a comparison operation:

  • greater than
  • less than
  • equal to
  • rises by
  • drops by

3. In the text field under Alert when, enter a value.

For example, you may want to "Alert when number of events [is] greater than 10".

Configure alert actions

Tell Splunk what to do once an alert is triggered.

1. Now set up how you want Splunk to notify you. You can combine any of these options:

  • Create an RSS feed
    • This creates a link to an RSS feed of alerts.
  • Send email
    • Enter one or more email addresses. Separate multiple addresses with a comma.

2. Next, if you want to include the search results in your alert, check Include results.

3. Finally, if you want to run a shell command when an alert triggers, enter the command under Trigger shell script. For example, you may want to trigger a script to generate an SNMP trap or call an API to send the event to another system. For more details see the page on scripted alerts.

Set up an alert on an existing saved search

You can take a saved search you've already created and turn it into an alert.

1. From the drop-down menu to the left of the search bar, choose Saved searches > Manage saves searches. This will launch the saved searches window.
2. In the table, locate the saved search that you want to turn into an alert.
3. Click enable in the Running column.

  • If you do not have permission to edit this search, the Running column will show *No*.
  • If there is already an alert defined for this saved search, it will either be Running or give the option to start it if you have the proper permissions.

4. To set up an alert, click the box next to Run this search on a schedule under Alert properties.

  • The options under Alert properties are the same described above for Schedule & Output.

Specify which fields to show

When you receive alerts, any fields included in your search are also displayed. Edit the saved search to change which fields show up in your alert.

  • To remove a field, pipe your search to fields - <field>. For example:

error starthoursago::01 | fields - sourcetypeSearch

This search keeps the sourcetype field from appearing in your alerts.

  • To add a field, pipe your search to fields + <field>. For example:

error starthoursago::01 | fields + clientIPSearch

This search adds the clientip field to your alerts.

You can add or subtract any number of fields -- just separate them with a comma: fields - <field1>, <field2> + <field3>, <field4>.

View alert history

The alert history page shows which alerts have been triggered since Splunk's last reboot. To access, click the Admin link in the upper right hand corner and select the Saved Searches tab. Your alerts show up in the Alert History column.

Previous: How Alerts Work    |    Next: Set up alerts via savedsearches.conf

Comments

No comments have been submitted.

Log in to comment.