The most requested information since my last Splunk(x) blog post was regarding the VMware environment. I would like to take a few moments to describe the Splunk(x) virtualization stack and the Splunk environment.
Our production VMware cluster is hosted in our private cloud at Equinix. The cluster consists of 8 ESX hosts with 12 cores and 96GB RAM for a total of 96 CPU cores and 768 GB RAM. Splunk(x) shares this environment with our production web infrastructure serving almost everything on splunk.com.
Splunk(x) has its own dedicated storage shelves in our NetApp FAS2040 composed of 44x 600 GB 15k SAS drives providing about 7,200 real IOPS to Splunk. With caching, we’re very close to 1,000 IOPS per indexer.
The Splunk(x) environment consists of 10 indexers, 3 pooled search heads, and a separate search head for overhead monitor displays. We have dozens of universal forwarders monitoring more than 30,000 sources 460 hosts. Splunk(x) is indexing more than 30GB/day. We have one heavy forwarder in our DMZ for catching Amazon EC2 data and one syslog catcher.
Splunk is eating web logs, Cisco switch logs, ShoreTel Director logs, Linux syslogs, Windows logs, and SalesForce.com data among other sources. We’re using our own app for F5 Networks, web intelligence app, app for enterprise security, app for VMware, and several other Splunkbase and custom apps.
I hope this overview of our Splunk(x) environment was helpful! As always, please feel free to post questions in the comments section or suggestions for future Splunk(x) blog topics.