Forums: SplunkAdministration: Alert Escalation Model

Hey guys,
We have a rotating on call escalation schedule. Is it possible to configure escalation within splunk? I.E. Alert user A, if they don't ack an alert, alert user B 10 minutes later. I don't expect it do to calendar calculations as that can be quite complex, but I'd like to write a bash script that runs once a week and updates our escalations based on our Google Calendar settings.

Thanks,
Todd

Not really, unless you write pretty much all the code yourself. Splunk alerts aren't acknowledgeable, they are just notification triggers. Once they fire, they call their script and are forgotten.

I know for us this is something we could really use, this way we can track what issues have been addressed and which ones are still pending. Would the plugin architecture allow me to write something like this?

It sounds like you want to feed your alert into a trouble ticket management system, or work request queue, or something like that.

I'm sure you can interface with Remedy, or Peregrine, or some other related system.

We forward our alerts to a home-written system which has human operators that managed the escalation and closure.

Sure you can. The Splunk alerts (or triggers) just call an arbitrary script, which you can use to call any other functions or notify some other system. It does get passed arguments including which Splunk Saved Search triggered it. The default Splunk script just sends an email, but you'd probably have it send to something else that will keep track of the state of the alert.

Just a note from Splunk product management - we've long kept workflow out of scope and instead focused on being easy to integrate with ticketing/workflow/service desk systems. There are so many great ticketing systems out there it seems like a wasted effort to built this functionality into Splunk. It's something we're always interested in hearing about though so if you have a particular use case where you strongly believe Splunk needs enhanced workflow functionality instead of integrating, please send details as an enhancement request to support@splunk.com.