By Splunk December 17, 2010
In certain contexts that could be a poorly chosen pickup line. In a distributed Splunk environment, it’s a real question which deserves more than eye rolling. Did you know, Splunk ships with a set of Status dashboards which could mean the difference between flying blind and having some valuable administrative insight into
- who is using Splunk?
- what problems Splunk indexers are encountering?
- when alerts are kicking off?
- why Splunk searches are slow?
- where Splunk is expending the most resources?
Maybe you already knew that. But did you know, the Status dashboards may not be entirely complete or accurate?
Because we want your first experiences with Splunk to be powerful and simple, the shipping configuration is built for a standalone Splunk instance. Everything is self-contained on the initial install so you can begin searching, reporting and building dashboards unfettered by concerns over configuration and architecture. As you progress beyond the out-of-the-box experience to deploying a production implementation, features like the Status dashboards need to be updated to account for a more sophisticated environment.
The Status dashboards live in the Splunk Search App, and provide visibility into user behavior, index/server health, data progress and more.
On a standalone Splunk instance, the dashboards will populate automatically. In a distributed search environment, however, they are not configured to query all Splunk indexers, only the distributed search head. This may not be apparent because some dashboards will still contain information, charts will render, they mostly look fine. However, content on indexing errors and inputs activity will be incomplete or missing altogether. The problem is more acute if your distributed search head queries other search heads.
The good news is this is quite easy to fix. The dashboards are driven by searches which refer ultimately to a single macro. This macro has the local server hard-coded. By removing the local server, all dashboard searches will distribute and aggregate content.
The macro can be changed by going to Manager > Advanced search (left column, towards bottom) > Search macros > audit_searchlocal(1), then deleting “splunk_server=local” from the macro definition. Remember to save the change. You may need to wait a few minutes for the change to take effect.
We will work on fixing this in future versions of Splunk. As of version 4.1.6, this change still needs to be made manually. Once this change is made, please remember to consider it during upgrades. Editing the macro creates/edits the file $SPLUNK_HOME/etc/apps/search/local/macros.conf with the new macro definition. If Splunk ever decides to change the default definition of the macro, the local definition will continue to override it. So when we fix this in the future, it’s a good idea to remove the edited definition from local/macros.conf.
In conclusion, the next time someone asks, “What’s your status?”, you can say, “They’re doing great!”
----------------------------------------------------
Thanks!
Vi Ly