One of the more recent use cases for which we’ve begun using Splunk(x) is website availability monitoring. It’s not enough to know that our web server is up—we want to know that it’s able to properly serve requests within tolerances of response time, bytes received, and HTTP status code expected.
Pinging the web server will tell you if it’s responding but not if it’s still serving web pages.
Access logs will tell you that the web server is serving web pages but gives little insight into the user experience.
Monitoring processes on the webserver will tell you if the server daemon is running but again sheds little light on availability.
Enter the Pinger add-on for Splunk.
With a few quick configuration lines, Pinger allows Splunk to monitor multiple web resources to determine connection time, response time, bytes served, and HTTP status code returned. A real-world example of the app’s utility is our web team’s monitoring requirements for Splunk.com.
Among other things, the web team wants to know if / on Splunk.com is serving content within an expected range of 30-50 KB, serving that request in under 1.5 seconds, and retuning a status code of “200”. We can determine this by scheduling the following realtime search:
index=main sourcetype=sla_heartbeat domain="www.splunk.com" resource="/" ( status!=200 OR request_time > 1.5 OR bytes < 30000 OR bytes > 50000)
We probably don’t want wake-up calls every time this search returns results (though we would want to review the incident later) so we can set alert criteria to only fire the alert if number of results is greater than five in a one-hour period.
Another cool thing about the Pinger app is that we can set the location from which the monitoring is occurring. Using this feature, we can define different alert thresholds for different geographic regions. Using the Splunk.com example, 1.5 might be the highest acceptable response time in the U.S. but we might tolerate a 2.0 second response time in the U.K.
Configuring the Pinger app is quite simple. To start monitoring a web resource, just create pinger.conf in the local directory of the app and add these stanzas (change the values to reflect your property)
[default] location = My Office [example_com] label = My Website host = www.example.com resource = /
More examples and documentation for pinger.conf can be found in default/pinger.conf.spec and default/pinger.conf.example.