Embracing Excellence: Splunk Best Practices at Your Organization

You count the quick reference guide as your right-hand man and the Splunk Tutorial helped you and others get up and running. You’ve watched all of the videos posts and can recite the past sixty-two five Splunk Talk podcasts verbatim. You’re registered for .conf2012. You’d planned on attending your local area SplunkLive! but a production issue kept you chained to your workstation until 11pm that night.

And you still want more.

You know, deep down, you’re only using 15% of Splunk’s potential. You also know you’re not alone in the company. How many of your co-workers are innovating cutting-edge solutions? What awesomeness are others crafting? How can you collectively take Splunk to the next level?

The answer is to establish a Splunk Center of Excellence (CoE).

The purpose of a COE is simple – to provide Splunkers an informal venue in which they can discuss ideas, diagnose challenges, share innovations and network with peers.

In order to create and maintain a successful Center of Excellence practice there are few requirements to consider.


The key ingredients to any thriving Center of Excellence are passionate individuals with an interest in continuous improvement. Do you have to be a “Duchess of Dashboards” or a “Sultan of Search”? Nope.

The only requirement is a drive to share knowledge and foster collaboration. The most successful CoE groups have multiple organizers or follow a floating schedule of organizers. The responsibilities of the organizer include;

  • Communicating the agenda and schedule to participants
  • Coordinating and lining up Resources / Presenters
  • Delegating a “second-in-command” to take the lead if a conflict arises

For sanity purposes it is highly recommended that another individual “volunteer” for the role of scribe. You might also consider recording it, using Evernote, Quicktime or other tool on your laptop. Effectively moderating the event is usually more than enough for any one person.

One of the most beneficial results of CoE meetings is knowledge of who the subject matter experts are in your organization. One of my customers had been struggling with a particularly difficult data source for over a week. At the next CoE meeting he was introduced to another group which had solved the same challenge months earlier.


Most groups consider one monthly meeting, lasting no longer than 60 minutes, to be the “sweet spot.” Successful groups meet on a consistent schedule such as the third Thursday of each month or some other easily scheduled period. And the agenda will always start with 5 minutes of introductions, followed by 20-30 minutes of presentation. And the meeting will always conclude with “open discussion” items for 10-15 minutes and a preview of what topics will be covered next month.

Nothing kills a fledgling Center of Excellence faster than two consecutive cancelled meetings.


While some organizations are able to physically meet at a common location, that option is becoming increasingly rare. Successful Centers of Excellence rely upon “meeting software” such as WebEx, Adobe Connect, Microsoft Lync and other software options. A persistent conference call phone number and access code are also elements of success. Make sure the organizer and at least one other participant (often the “scribe”) are aware of any special “host” or “moderator” codes required for the call to take place.


The entire purpose of the Center of Excellence is to create an environment that fosters knowledge-sharing and internal “best practices.” If there is no centralized mechanism to “share” the information, you’re putting a lot of faith in participants’ note-taking skills. Whether it’s your corporate instance of SharePoint, Stream Works, Box or Dropbox you’ll need (to paraphrase George Carlin) “a place for your stuff.”

Some of the best Centers of Excellence have created internal Splunk Wiki content or “run books” unique to their own environment. This saves a great deal of time in getting somebody new to Splunk, up and running.

One of our CoE customers has created an internal Wiki page which lists all required information for new data, instance architecture, on-going projects and corporate mandates based entirely on their monthly meetings. Through collaboration they have created a “master” document about their own Best Practices. This allows them to on-board new groups into the Splunk environment, from Dev to Production in less than 4 business days.

Another customer created an IM “list” of Splunkers in their corporate environment. This simple step allowed them to network internally before going to Splunk Support. As a result their number of Cases dropped by over 20%.


One of the biggest challenges in getting any CoE off the ground is finding enough material to warrant an hour-long meeting once per month. Fortunately that is seldom a problem with Splunk. There is a myriad of topics to discuss. Some examples include:

Check out our Slideshare pages for other ideas. The list goes on. If you’ve heard of a new Splunk App you’d like to know more about, feel free to ping your Splunk contact. Solution Engineers are always happy to show off our shiniest new toys to customers. Keep an updated list of past agenda items and future topics. As changes occur in your organization and new challenges arise, the topics can be easily modified to accommodate your requirements.

Follow Through

Congratulations! The inaugural Center of Excellence had great attendance, fascinating content, and lively discussions. But you’re not quite done yet.

Another crucial element common to successful Centers of Excellence is the prompt distribution of minutes from the meeting. Be sure to capture any action items, unresolved questions or even screenshots. Always conclude the minutes with a reminder of the next meeting and the topic to be addressed.

Make it Fun!

I know work isn’t supposed to be fun – why else would they call it “work”, right? But Splunk is the exception to that rule.

Have contests among your peers. There’s nothing like a little friendly competition to get the creative juices flowing.

  • Who can create the most complex Search string?
  • Which group has created the most useful Application?
  • Focus on a new or challenging business requirement and determine who can create the most effective solution
  • See which team can come up with the most ways to “re-use” a particular data source
  • Who can bring the largest number of Splunk beginners to the next meeting?
  • Dashboard Beauty Contests – just make sure it doesn’t turn into an episode of “Toddlers & Tiaras.” That gets ugly…

Hit up the Splunk store for some awesome t-shirts,coffee mugs or hats as prizes. We’ve got a lot of schwag we’d like to share with the winners.


The majority of our customers experience a similar growth rate with Splunk; initial adoption in a smaller group, followed by rapid expansion to other areas of the business as additional benefits are realized. As the scope and number of users grows, so too does the value of a Center of Excellence. The monthly meetings are also an excellent venue for the curious to learn a little more about what is being done internally with Splunk.

With just a little preparation and the right attitude, you can get your Center of Excellence off the ground and soaring in no time. Keep it simple, make it informal and have fun.

Chris Bauer

Posted by