Update 2016-10-31: Thank you Doug Brown and Hal Rottenberg for collaborating over the weekend to clarify this post! -eric grant, Community Manager, Splunk Community
Update 2016-10-28: There have been a number of questions from readers about the specs recommended in this post. Splunk is working with the author to clarify the numbers, and stands by the author’s right to make performance recommendations, based on his experience, that differ from our official requirements. -eric grant, Community Manager, Splunk Community
(Hi all–welcome to the latest installment in the series of technical blog posts from members of the SplunkTrust, our Community MVP program. We’re very proud to have such a fantastic group of community MVPs, and are excited to share what we learn from them. –rachel perkins, Sr. Director, Splunk Community)
Hi, I’m Doug Brown, Information Security Analyst at Red Hat, and member of the SplunkTrust.
Over the last few years I’ve spoken with a number of Enterprise Security customers from different regions, and I’ve received mixed feedback about their deployments. The good news is that there are some easily-avoidable common pitfalls, and by being aware of these before engaging Splunk Professional Services, hopefully you’ll be able to derive the greatest value possible from Enterprise Security.
The most common issue I hear about is performance. There are a number of compounding reasons why this can be the case. Although the reference hardware for indexers suggests a minimum disk performance of 800 IOPS, the requirements for a production Enterprise Security deployment
is much greater. From my experience, I recommend the high-performance specifications for indexers. Splunk makes hacking at big, dodgy datasets look trivial, but there’s no avoiding the significant load required to perform these tasks. As such, ensure you have the necessary spec before Splunk Professional Services comes on site, because Splunk Enterprise, let alone Enterprise Security, will never fly without the necessary hardware. The de facto way we stress test IOPS is using bonnie++ (http://www.coker.com.au/bonnie++/). Just be sure to let your storage admins know before running the test.
Disk isn’t everything though–the operating system does make a difference. Your organisation might consider itself to be a “Windows shop”, but using an enterprise-grade Linux distribution is essential to good performance. Ensure that Transparent Huge Pages is turned off, and the default ulimits have been increased. Also, virtual doesn’t have to equal poor performance when it comes to Splunk. It’s true that certain virtualised configurations can be sub-optimal, but a well configured enterprise-grade hypervisor on good hardware, with LUNs presented from decent
storage can be just as good as bare metal, and provide greater flexibility.
Before Splunk Professional Services arrive, be sure to upgrade your existing Splunk Enterprise servers across your infrastructure. It will have to be done before Enterprise Security is installed anyway, so you may as well do it yourself. In this way, your time with Professional Services can be better spent on the deployment itself.
Well ahead of time, it’s important to identify your organisation’s obscure, custom, but important data sources that Enterprise Security needs to “see” (have mapped to the CIM). These are sources of information for which there isn’t an existing certified TA app on Splunkbase. For example, if your organisation has a custom SSO technology, speak with the subject matter experts to find out the key security-relevant events, and document them ready for PS. Better yet, write your own using Add-On Builder, then be awesome and release them under and Open Source license on Splunkbase.
Enterprise Security is all about enrichment, and underpinning that is the identity and asset lookups. Without this, the value you can derive from ES is significantly constrained. Keep in mind that PS won’t be able to produce these for you, as they don’t know your organisation. Find someone internally that can help you programmatically produce these lookups according to the format specified in the documentation (docs.splunk.com/Documentation/ES/latest/User/AssetandIdentityLookupReference). This will likely require a significant investment of time and energy to munge and hack data from from various sources, but it’s well worth the effort. Don’t worry too much about populating fields you don’t care about or don’t have a source of truth for (‘priority’ being the exception). In fact, less can be more, as those lookups have to be propagated to the search peers (indexers) at search time. If they’re too big, it can cause performance issues, so try to keep them a reasonable size (ideally below 10MB each).
If Enterprise Security is to be installed in a search head cluster, be sure to have ready a fresh Linux virtual machine that PS can use for staging. There are no spec requirements for the staging machine, anything will do. Finally, if using a proxy, ask Professional Services for the list of Enterprise Security’s threat feeds so you can configure your proxy settings beforehand.
Good luck with your preparation and be sure to check back here soon for the next installment in this series where we’ll discuss what to do when Splunk Professional Services arrive.
Thanks to Rachel and Alison Perkins, Simon Duff, Russ Uman, Joshua Rodman, and the many SplunkTrust members whose feedback helped shape this post. May the advice here help improve the security of organisations great and small.