The time has finally come for us to bring Splunk(x) to the far reaches of the
galaxyworld. I got a call from our Fed team requesting Splunk(x) monitors in our Bethesda, MD office so they could show off how we use Splunk to our public sector opportunities and customers. Always eager to go take awesome photographstravel for work, I happily obliged and you can see the results after the break.
As I was summarizing the effort to the awesome IT and engineering folks who helped make this happen, I realized we had some pretty important takeaways from our experience of setting up Splunk(x) monitors at a remote location. I also felt inspired to write this blog sharing those experiences!
- More search heads are better. We’ve been running our San Francisco monitors off one search head since we stood them up in February. This is partially because of the complexity of the architecture. We have our SF search head on the same pool as the search heads that serve interactive users. We ran into a performance wall at .conf because of this. By allocating more search heads, we can tear down this wall and present data more efficiently.
- Monitors don’t need to be on the pool. Because we’re requiring always-on monitors to pull summarized data rather than hit indexes directly, we can remove them from both the search head pool and unmount the bundles. This makes the dashboards paint far more quickly than our previous monitors.
- Monitor search heads can still be virtual. With Splunk(x), we’ve caught ourselves in the middle of the “to virtualize or not to virtualize” argument often. The answer is not easy; it depends on how you’re using Splunk, how much is being indexed, and several other factors. For the search head under overhead monitors hitting only summary data; however, we say “yes we can” to virtualization!
- Monitor search heads can be secure. A major concern with these search heads was the ability to arbitrarily access Splunked data. They’re always on, so what stops someone from closing a dashboard and hitting a flashtimeline view to see all our SalesForce.com data? Well, because the monitor user can only access the summarized data and because only admins can log in from Active Directory, we’ve successfully mitigated this concern.