Stories from Customer Zero: Building a Scalable RBAC Methodology on Splunk

Here at Splunk we take pride in our tenacity to disrupt conventional approaches. The following exposition captures our implementation and current thoughts about being Customer Zero of Splunk. Our recommendations will be formalized and packaged for easier consumption through our Best Practices communications.

The Challenge

Getting your data into Splunk is only a first step. Making all that data in your Splunk ecosystem available to all your users in an actionable form is a bit more involved. As you drive to maximize your investment getting actionable intelligence out of your machine data, use cases will typically expand to include other data sets. As deployments grow, design issues can arise.

It’s a constant balancing act, struggling to adequately secure data and the derived knowledge that sits on top of it, against the need to make the data available to those who need to work with it.

Building for scale always brings significant challenges. There are four key concerns we were trying to address when we started this effort:

  1. Adequate data security
  2. Adequate knowledge security
  3. Adequate permissions restriction
  4. Can the solution scale properly

Data Security

This is a major issue, but an easier one to address. How do you restrict access to secure data? It’s easy to lock that index to a single role, but now you're running at scale... How do you keep track?

Knowledge Security
Lookups, database connections and API queries cannot be governed by index restrictions, so how do you manage access on this restricted knowledge?

Permissions Restriction
Depending on how developer-friendly your Splunk environment is, it can become quite challenging to manage all the various permissions. How do you keep track when things get large and complex?


Great that you created roles and you're dictating access through Splunk, but what happens at scale when you've got more data access requests than hours in a day? How do you future-proof your system to handle growth at scale? It’s a complicated knot that can be difficult to disentangle.

Disentangling the Knot

We started with what we knew:

  • Some indexed data needed to be secured
  • Those datasets needed to live in their own indexes (Securing Splunk Enterprise documentation)
  • Some knowledge objects needed to be secured
  • Certain permissions needed to be revoked

Solution requirements:

  • Whatever process we use must scale
    • Must adopt a standardized naming convention
    • Must use as much automation as practical, reducing touch points
    • Data type determines security controls
  • Solution needs to be broad enough to address all use-cases
    • Application & Knowledge Object Security
    • Indexed Data Security
    • User Access Privilege Levels

So we started figuring out how to map it all out.

Our Solution

We started with the structuring mapping of access and privilege. We wanted to avoid the dynamic that had created unfavorable issues for us in the past, so we decided to separate all logical aspects of access.

This meant a user’s assigned roles and groups would dictate index access, application access, and user privilege.

The "why" for this approach came down to the lowest common denominator for logical groups of security. Our indexes, by design, were structured around likeness, retention, and sensitivity. Knowledge object security is a little more tricky—while objects are always associated to an application, end users may set permissions to global, or even clone them into other areas. The only true way to secure knowledge objects because of that is twofold: first, enable application access on an "as needed" (by sensitivity) basis; and second, monitor and enforce knowledge access to the app level for those sensitive applications.

Once we decided to map things out in this way, the challenge of designing templates for the role mapping was the key remaining task. Many of you will have corporate template guidelines for AD groups just as we do.

The following table breaks down the template we settled on. Obviously this exact template might not be what you want to use, but we would recommend following this basic methodology.


Group Description

Group Convention

Index Security Group  


Application Security Group 


Privileged Access Security Group  



Variable Description


The full index name (best practices further dictate the use of a standardized naming convention)


This is the folder name inside the etc/apps directory in Splunk

APP <access_type>  

This value reflects the rights to a given app, we use RO and RW


Functional unit inside a given org

PERM <access_type>

Privilege and capability access , we use USR (standard splunk user role) PWR (standard power user role) and DEV (a blend of power and admin roles)

From here we created roles for each group for a one to one mapping.

srchIndexesAllowed = test
srchIndexesDefault = 

The last problem to overcome with this was how to manage changes to this schema going forward. What we decided on was to use a hidden app to manage the roles and rolemap stanzas in the authorize and authentication conf files.

After that, all we had to do for updates was edit our app and check the code into git, then deploy the changes. From here we need to get the corresponding AD (or if you are using another identity service) groups created and assign approvers for granting access to each one.

Unfortunately, this is where the real footwork comes into play—a list of all users who should have access needs to be put together. We compiled this list by working with the data owners (for indexes that need medium security), and then by the InfoSec team (for indexes that contain sensitive information). Going forward, the data owners and InfoSec team would be listed on the group approval list (again by security need of a given index).

Finally, everything was in place and we could hand off the access mapping guide to IT end user services for L1 support; washing our hands of having to manage access requests.

Final Thoughts

You can put whatever controls in place, but they must be followed to be effective. In a future blog post, we’ll talk about how we monitor data access (Data Security Monitoring) to insure compliance with these controls and to identify and notify on security breaches.

Alex Cain

Posted by