Hardware capacity planning for your Splunk deployment

Hardware capacity planning for your Splunk deployment

Splunk is a very flexible product that can be deployed to meet almost any scale and redundancy requirement. However, that doesn't remove the need for care and planning. This article discusses high level considerations for Splunk deployments, including sizing and availability.

After you've worked through the general layout of your Splunk search topology, the other sections in this document can explain more thoroughly how to implement them, along with the formal Admin guide for Splunk.

Reference Hardware

Let's consider a common, commodity hardware server as our standard:

• Intel 64-bit chip architecture
• Standard Linux or Windows 64-bit distribution
• 2 CPU, 4 core per CPU, 2.5-3Ghz per core
• 8GB RAM
• 4x300GB SAS hard disks at 10,000 rpm each in RAID 10
  • capable of 800 IO operations / second (IOPS)
• standard 1Gb Ethernet NIC, optional 2nd NIC for a management network

For the purposes of this discussion this will be our single server unit. Note that the only exceptional item here is the disk array. Splunk is often constrained by disk I/O first, so always consider that first when selecting your hardware.

Performance Checklist

The first step to deciding on a reference architecture is sizing - can your Splunk handle the load? For the purposes of this guide we assume that managing forwarder connections and configurations (but not their data!) to be free. Therefore we need to look at index volume and search load.

Question 1: Do you need to index more than 2GB per day?

Question 2: Do you need more than 2 concurrent users?

If the answer to both questions is 'NO' then your Splunk instance can safely share one of the above servers with other services, with the caveat that Splunk be allowed sufficient disk I/O on the shared box. If you answered yes, continue.

Question 3: Do you need to index more than 100GB per day?

Question 4: Do you need to have more than 4 concurrent users?

If the answer to both questions is 'NO', then a single dedicated Splunk server of our reference architecture should be able to handle your workload.

Question 5: Do you need more than 500GB of storage?

At a high level, total storage is calculated as follows:

  daily average rate x retention policy x 1/2

You can generally safely use this simple calculation method. If you want to base your calculation on the specific type(s) of data that you'll be feeding into Splunk, you can use the method described in "Estimating your storage requirements" in this manual.

Splunk can generally, including indexes, store raw data at approximately half the original size thanks to compression. Given allowances for operating system and disk partitioning, that suggests about 500GB of usable space. In practical terms, that's ~6 months of fast storage at 5GB/day, or 10 days at 100GB/day.

If you need more storage, you can either opt for more local disks for fast access (required for frequent searching) or consider attached or network storage (acceptable for occasional searching). Low-latency connections over NFS or CIFS are acceptable for searches over long time periods where instant search returns can be compromised to lower cost per GB. Shares mounted over WAN connections and standby storage such as tape are never acceptable.

Beyond 100GB/Day

If you have requirements greater than 100GB/day or 4 concurrent users, you'll want to leverage Splunk's scale-out capabilities. That involves using distributed search to run searches in parallel across multiple indexers at once, and possibly load balancing the incoming data with auto load balanced Splunk forwarders.

Also, at this scale it is very likely that you'll have high availability or redundancy requirements, covered in greater detail below.

Question 6: Do you need more than 300GB/day of daily indexed volume?

If you do not - i.e. you are between 100GB/day and 300GB/day - you should be able to have multiple dual-purpose Splunk boxes that are searching across each other.

example of a search user searching on one Splunk instance and having their search distributed to other instances

Additional Considerations

• you can use a third party load balancer to assign users to different Splunk instances
• the recommended best practice for Splunk forwarder management is to use auto load balancing
• you can use Splunk deployment server to propagate Splunk apps and user preferences between instances
• if you need more than 4 concurrent search users 'per server' this deployment is not appropriate
  • for example if you have 2 reference servers but need more than 8 search users

Beyond 300GB/Day

For deployments of 300GB/day or larger, consider a three tier Splunk deployment. In this model, search is separated from index by creating Splunk search heads, or instances of Splunk that only do searching. That allows for more efficient use of hardware, and to scale search usage (mostly) independently of index volume.

Example Splunk distributed topology. This example could handle up to 400GB/day and 8 concurrent search users for common use cases.

Dividing Up Indexing and Searching

At daily volumes above 300GB/day, it makes sense to slightly modify our reference hardware to reflect the differing needs of indexers and search heads. Search heads do not need disk I/O, nor much local storage. However they are far more CPU bound than indexers. Therefore we can change our recommendations to:

Search Head

• Intel 64-bit chip architecture
• Standard Linux or Windows 64-bit distribution
• 4 CPU, 4 core per CPU, 2.5-3Ghz per core
• 4GB RAM
• 2 300GB SAS hard disks at 10,000 rpm each in RAID 0
• standard 1Gb Ethernet NIC, optional 2nd NIC for a management network

Given that a search head will be CPU bound, if fewer, more performant servers are desired, adding more and faster CPU cores is best.

Note: The guideline of 1 core per active user still applies. Don't forget to account for scheduled searches in your CPU allowance as well.

Indexer

• Intel 64-bit chip architecture
• Standard Linux or Windows 64-bit distribution
• 2 CPU, 4 core per CPU, 2.5-3Ghz per core
• 8GB RAM
• 8 300GB SAS hard disks at 10,000 rpm each in RAID 10
  • capable of 1200 IO operations / second (Iopps)
• standard 1Gb Ethernet NIC, optional 2nd NIC for a management network

The indexers will be busy both writing new data and servicing the remote requests of search heads. Therefore disk I/O is the primary bottleneck.

At these daily volumes, likely local disk will not provide cost effective storage for the time frames that speedy search is desired, suggesting fast attached storage or networked storage. While there are too many types of storage to be prescriptive, here are guidelines to consider:

• indexers do many bulk reads
• indexers do many disk seeks

Therefore...

• more disks (specifically, more spindles) are better
• total throughput of the entire system is important, but...
• disk to controller ratio should be higher, similar to a database

Ratio of indexers to search heads

Technically, there is no practical Splunk limitation on the number of search heads an indexer can support, or the number of indexers a search head can search against. However systems limitations suggest a ratio of approximately 4 to 1 for most use cases. That is a rough guideline however; if you have many searchers compared to your total data volume, more search heads make sense, for example.

Summary of Performance Recommendations

Daily Volume	Number of Search Users	Recommended Indexers	Recommended Search Heads
< 2GB/day	< 2	1, shared	N/A
2GB/day to 100GB/day	up to 4	1	N/A
200GB/day	up to 8	2	1
300GB/day	up to 12	3	1
400GB/day	up to 8	4	1
500GB/day	up to 16	5	2
1TB/day	up to 24	10	3

Note that these are approximate guidelines only. You should feel free to modify based on the discussion here for your specific use case, and to contact Splunk for more guidance if needed.

High Availability and Data Redundancy

Many Splunk deployments require some form of redundancy, either to protect the data from loss or the search service from outage - and sometimes both. In general Splunk's solution to this problem is a straightforward matter of data duplication, however we will look at three specific deployment possibilities.

Data Duplication

The easiest method of ensuring data will not be lost is to have two original artifacts made by cloning data coming from Splunk forwarders.

In this approach, the data is duplicated and available instantly, should you need to cut over to the stand-by Splunk instance. Note that while you can simply have one Splunk forward to the next Splunk (as shown here for the offsite location) to save on network usage, there is a risk on hard shutdown of the last few events not being sent on. If that is acceptable, the topology can be even simpler.

High Availability

The goal of a high availability deployment is both data survivability and service uptime. To accommodate this kind of deployment, you need to duplicate both the data and the physical hardware providing service, not unlike other web based applications. Also, redundancy needs to be considered for all three tiers of service - splunkweb searching, splunkd indexing and forwarding.

In this topology, there are two data complete functional groups. In the picture both groups are servicing search requests to optimize hardware costs; the second infrastructure could be idle to ensure neither disruption nor degradation of search services.

Things to note about this topology

• search heads can be load-balanced to allow for user redirection should one go down
• as long as there is at least one indexer up in a clone group, the dataset remains intact
  • so long as the surviving indexers can handle the load; it is recommended to stop searching against a degraded group to ensure it doesn't fall behind

Performance Considerations

Splunk has three primary roles - indexer, searcher and forwarder. In many cases a single Splunk instance may two or all three roles at once. All have their own performance requirements, and bottlenecks.

• indexing, while relatively resource inexpensive, is often disk I/O bound
• searching can be both CPU and disk I/O bound
• forwarding uses very little resources, and is rarely a bottleneck

As you can see, disk I/O is frequently the limiting factor in Splunk performance, and deserves extra consideration in your planning. That also makes Splunk a poor virtualization candidate unless dedicated disk access can be arranged.

CPU

• allow 1 CPU core for every 1MB/s of indexing volume
• allow 1 CPU core for Splunk's optimization routines for every 2MB/s of indexing volume
• allow 1 CPU per active searcher (be sure to account for scheduled searches)

Disk I/O

• assume 50 Iopps per 1 MB/s of indexing volume
• allow 50 Iopps for splunk's optimize routines
• allow 100 Iopps per search, or an average of 200 Iopps per search user

Memory

• allow 200-300MB for indexing
• allow 500MB per concurrent search user
• allow 1GB for the operating system to accommodate OS caching

Total Storage

• allow 15% overhead for OS and disk partitioning
  • on this system there is ~500GB of usable storage
• conservatively Splunk can, including indexes, compress original logs by ~50%
  • compression rates vary based on the data

Based on these estimate, this machine will be disk IO bound' if there are too many active users or too many searches per user. That is the most likely limitation for this hardware, possibly followed by CPU if the searches are highly computational in nature, such as many uses of stats or eval commands in a single search.

Applied Performance

With the information above, it is possible to estimate required hardware for most Splunk use cases by considering the following:

• the amount of daily indexed volume (disk I/O, CPU)
• the required retention period (total storage)
• the number of concurrent search users (disk I/O, CPU)

Note that not all search users consume the same amount of resources. While their are in depth guides for search cost analysis available here, consider these very rough guidelines.

• dashboard-heavy users trigger many searches at once
• dashboards also suggest many scheduled searches
• searching for rare events across large datasets (e.g all time) is disk I/O intensive
• calculating summary information is CPU intensive
  • if done over long time intervals, can also be disk I/O intensive
• alerts and scheduled searches run even if no one sees their results

What does that mean in real life?

• executive users with many dashboards and summaries require both CPU and disk I/O
• operations users searching over recent and small datasets require less resources
• forensic and compliance users searching over long timeframes require disk I/O
• alerting and scheduled searches over short timeframes are inexpensive; over long timeframes potentially very expensive.