By Splunk January 13, 2011
How do I staff for Splunk? A lot of people ask this question and there is not a great deal of information about the topic. While Splunk can be easy to use and maintain, proper care must be taken to ensure a healthy running Splunk instance. In this posting, I will detail the most important items with respect to staffing. First, let’s break this down into 2 topics.
- Required skill sets to administer and maintain a Splunk installation
- Resource levels required for different installation sizes.
Skill Sets
Let’s start out by considering Splunk specific skills for an administrator. We recommend that a Splunk administrator starts out by attending our user course and administration course. These courses will provide a new administrator the necessary skills to perform very basic support for a’turn-key’ Splunk installation. There are certain activities not covered in these courses, so I must emphasize that this is the minimal requirement to perform basic administration and support duties. Ideally, attending the additional offerings (deploying, developing, searching & reporting) are preferred. These additional courses are tailored towards specific Splunk functions. People who attend a complete Splunk track or certification, will generally experience higher return on value from their installation. This is because they are able to develop new uses cases that provide value well beyond the initial purchase proposition.
Okay, so enough about Splunk courses and how important they are. Let’s look at resume items that are important for maintaining Splunk:
- System Administrator experience: multiple years for the installed Operating Systems
- Computer Networking proficiency
- Internet/Web technology proficiency
- Scripting/coding experience is preferred
An individual with the above skills will have sufficient qualifications to become a Splunk administrator.
How many people do I need?
Determining staff size depends on these primary factors:
- Daily Indexing Volume
- Daily Search Volume
- Number of Data Sources
- Number of Users
Daily Indexing Volume – how much you index will dictate how many servers you may need to maintain. In most cases, the Splunk administrator maintains the hardware running Splunk. It can be stated simply: expect to have some level of maintenance duties with respect to all systems that send or receive Splunk data.
Daily Search Volume – if a Splunk system is only used a few times a day by a few people, there is minimal overhead to managing searches. However, if a system has dozens of users that run many searches each hour, this typically requires a Splunk administrator to monitor and maintain searches on a daily basis. Scheduled searches (alerts) and reports become production operation tools in these installations. Therefore, they require a production level administration team.
Number of data sources – simple inputs such as a single syslog source are generally easy to maintain. These inputs rarely change in format thus requiring minimal maintenance. When many data sources are used, more complex input methods get introduced such as scripted inputs. These complex inputs may require multiple adjustments to the way Splunk ingests, enriches, and returns the data.
Number of users – this is important although it’s not as important as the search volume. The number of users will drive the total volume as well as amount of resources required to sustain concurrent usage. Out in the working Splunk world, I’ve found that some systems have 100’s of users and it’s only a few that drive the majority of volume. This always ties into the use case, as that dictates the level of searching and alerting. In the end, one needs to factor in both of these with a primary focus on total search volume.
Now that we have an idea of what drives the number of staff members required, let’s look at some generalized numbers. These are simply guidelines and there are exceptions and qualifications:
- 1 Part Time Splunk resource: Capable of supporting a single instance of Splunk that has a few data sources and users (0-50 GB/day of indexing volume)
- 1 Full Time Splunk resource: Capable of supporting a multi-instance Splunk deployment that has many data sources and dozens of users (10-1000 GB/day of indexing volume)
- 2 + Full Time Splunk resources: Capable of supporting a multi-instance Splunk deployment that has hundreds of users and many data sources (100-1000+ GB/day of indexing volume)
So what about the extreme scenarios for high volume searching or indexing? A very well trained administrator could maintain many Splunk deployments that total 5+ TB/day of indexing volume. However, a lower volume (100 GB/day) but extremely highly utilized system (100k searches per day) might require the full attention of the same administrator.
In the end, Splunk is not difficult to support and maintain with respect to most software, and your Splunk admin life can be made easier with proper training.
----------------------------------------------------
Thanks!
Simeon Yep