Forums: SplunkReporting: Detecting the absence of a forwarder

I apologize in advance for the double post as I've seen this answered somewhere else, but I can't find it with search. We had an issue today where a REST service we monitor with a modified version of the webping APP stopped returning content. We didn't receive any errors on response codes or timeouts, we just stopped receiving data. Would it be possible to get an example query that would set off an alert if a host or a source receives no input for over 10 minutes? This will also help us find issues when we stop receiving data from our forwarders.

Thanks,
Todd

http://www.splunk.com/base/Deploy:HowToFindLostForwarders

Ok, so now I have another question. I have run the following query.

| metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

This returns to me 4 hosts. The hostname has been changed on the forwarder after it has forwarded data to our splunk instance. For instance, one of our server names is "test1". I don't care if this forwarder isn't receiving data, but I want to capture all future forwarders. I thought I could remove this data by doing the following.

host="test1"|delete

This appears to delete all the data, but does not remove it from the internal index. How can I clean up this index to remove all internal data so these old hosts no longer appear?

Thanks,
Todd

This will probably be in 4.0.4 or 4.0.5.

I tried this again in 4.0.4. It says it deletes the records from the internal index, but when I perform the query they still appear. Is there something else I need to run to delete the internal index data?

Thanks,
Todd

You must grant yourself the "can_delete" role or the "delete_data" capability. No, the admin user does not have that right by default, by design.