NOTE: There is a new and updated post on this topic located here.
Over the past year, Splunk has increased it’s footprint for installations on Amazon EC2. Along with this, come questions about best practices and recommendations for deploying in a cloud environment. In this post, I’ll provide some guidance around deploying Splunk on EC2. It is important to note that the search and indexing load will dictate the hardware requirement. The following link contains the appropriate guidelines for sizing: HERE
Let us first review our ‘reference’ server configuration for a deployment that indexes 10-100 GB per day:
- 8 cores (2 quad core, > 2.5 GHz)
- 8+ GB RAM
- RAID 1+0 Disk Configuration, disk speed > 10k RPM
For a deployment in EC2, we will consider the above reference numbers and adjust capacities based on the EC2 offerings. As of today, a deployment capable of indexing 10-100 GB per day might look as follows:
- 1 – Extra Large instance (Cluster Compute optional)
- 7+ GB RAM
- 4 or more EBS volumes, configured in RAID 1+0
- 1 EBS volume, leveraged for snapshots
- Availability Zone should overlap when possible
We must address the topics of preferred instance type/class and storage configuration. The “XL” instance is the common starting point for most installations, as it provides ample resources for searching and indexing. While the Cluster Compute type is not required, Splunk should see increased performance from this server class. Although EC2 provides a couple storage options, EBS is ideal. This, in combination with RAID 1+0 (use at least 4 disks, maybe 6) will provide the appropriate levels of performance for search and indexing. An additional EBS volume is recommended to act as a snapshot store.
If you plan to launch Splunk in EC2, the above info should be a good starting point for you. If not, feel free to post on Answers, respond to this blog, or email support.