By Splunk March 25, 2010
Find them fast and get them under control.
Back in the day, when I wrote Java code for a living, one thing I was asked to find was a unique list of broken links in the company’s web-based management console. I wrote an http client, which recursively crawled all URIs starting from the home page and tracked the URIs with HTTP return code other than 200. It took me a few days to write/test/package it for a non-technical end user.
Today, finding broken links with Splunk and access logs can be done in mere minutes. There are plenty of web/UI testing tools available, but as some of our customers like to say “the truth is in the logs.”
The basic Splunk search simply makes use of the powerful stats command:
sourcetype=access_common OR sourcetype=access_combined status!=200 | stats values(uri) as "Broken Links" values(status) as Status by status | fields + Status, "Broken Links"
Or if you are simply looking for the file, instead of the entire URI:
sourcetype=access_common OR sourcetype=access_combined status!=200 | stats values(file) as "Broken Links" values(status) as Status by status | fields + Status, "Broken Links"
A slightly more advanced search makes use of the same stats command, but also incorporates a status description lookup. The lookup is transparent and ties the numeric status code to something more descriptive. Instructions to setup the status description lookup are available here.
sourcetype=access_common OR sourcetype=access_combined status!=200 | stats values(status) as Status values(file) as "Broken Links" by status_description | rename status_description as "Status Description"
Additionally, these searches can be simplified for succinctness. If you don’t care about relabeling/reordering columns or adding row separators, try this:
sourcetype=access_com* status!=200 | stats values(status) values(file) by status_desc
It’s not an exhaustive search of all links on your website, but a good, fast review of the visited links tracked in the http access log. No coding required. HTH!
----------------------------------------------------
Thanks!
Vi Ly