Forums: SplunkReporting: Scheduled query with dynamic time range

Hi all,
We're using splunk to monitor our flight systems. We want to fire an alert off to the person on call when a result returns any events for a give query. This will trigger a manual action where search and rescue is contacted by our on call person, and a rescue effort will begin. However, I can't simply execute my query for the last "X" time. I need to be able to cover all timespans, included an unexpected downtime of splunk. I think I need to be able to store when scheduled job was last executed and execute a query that covers the delta between my last executed time and the current time. I'm assuming this data is stored in some internal index, how could I get to it in my query? Here is an example of what I'm currently doing at a 5 minute interval.

current search:

source="C:\\logs\\AlertMonitor.log" "TIER TWO ALERT GENERATED"

Rather than the time always at -5m, I'd like to search from -(currentTime-LastExecuted) and run the job every 5 minutes. Is this possible?

Thanks,
Todd

You could try some trickery like:

source="c:\\logs\\alertmonitor.log" "tier two alert generated" [ inputlookup lastruntime.csv | tail 1 | fields earliest_time ] | head 1 | append [ stats count | eval earliest_time=now() ] | outputlookup lastruntime.csv

And then set your alert criterion to "number of events > 2"

You might need to seed the lastruntime.csv file by running:
{{| stats count | eval earliest_time=now() | outputlookup lastruntime.csv}

I haven't tried this, but I think it will work. If it doesn't because of locking of the lastruntime.csv file, you can work around that by having two staggered alternating jobs running almost the same search, each writing to a different lookup file, and reading from the opposite one's lookup file.

If you have to actually return the results, you could drop the "head 1" clause, though you'll end up with an extra result appended to the end (that carries the last run time). it's possible to get rid of that after the lookup table has been output though.

[Revised on Mon, 09 Nov 2009 18:09:34 -0800]

oh, and you should probably clear the earliest time in the saved search config too. Note that you can do some math on now() if you need to offset or overlap, e.g., if your regular search does a latest_time-5min, you'd probably really want eval earliest_time=now()-(5*60).

Thanks for the help. It's greatly appreciated. I'm sort of following you, but I'm not 100% sure I understand what's happening internally. Here is what I think is happening, please correct me if I've mistaken any of the steps.

1. Read the lookup file "lastruntime.csv"
2. Get the last record from the file
3. Get the field earliest_time
4. Pipe the earliest_time through head 1 (I'm not sure why we use this since there is only 1 result from the tail 1, can you please elaborate on this?)
5. Run the query "source="c:\\logs\\alertmonitor.log" "tier two alert generated" from the earliest_time in the lastruntime.csv
6. create a 2 column ouptut with count and earliest_time to now and dump it to lastruntime.csv

Now, I'm unsure how the time range on my query in the first part of my search with "source="c:\\logs\\alertmonitor.log" "tier two alert generated" will have it's time range updated by the input from the lastruntime.csv. Can you give me a bit more info on how that's substituted?

Thanks,
Todd

A subsearch as a search argument implicitly expands its results out to a search term. the "inputlookup" subsearch finally ends up with a result of "earliest_time=XXXX", which is provided as an argument to the search in which it is embedded. (More accurately, the results are implicitly piped through the "format" command" which converts them to a search term.).

This is also why "head 1" is needed, since the "tail 1" is not part of the main search pipeline.