Reflections on a Splunk developer’s journey : Part 1

It seems like only yesterday

…that I was writing my first Splunk App. It was the openness and extensibility of the Splunk platform that attracted me to this pursuit in the first place, and when I discovered the thriving community on Splunkbase (now called Splunk Apps / Answers), I just had to contribute. 12,000+ downloads across 9 different freely available community offerings later, I am feeling somewhat reflective. So in this 2 part blog series I want to share with you some of my lessons learned from developing and supporting Splunk community Apps/Add-ons (part 1) and then some musings on why you should consider developing Splunk Apps/Add-ons yourself and contribute to the Splunk developer ecosystem (part 2).

Some lessons learned

Keep it simple

Try to make the App/Add-on as simple as possible in terms of install, setup, configuration and navigability. I try to implement solutions with the least number of things for the user to do and know in order to get up, running and productive. You can still have complexity, but just make sure you abstract this from the users in an elegant, intuitive and robust experience.


It’s going to make your life and you user’s lives are lot simpler if you have concise, easy to follow documentation that covers everything you’d expect to encounter in the normal usage of your App/Add-on. My documentation is also continually evolving as I respond to questions from users, add new features and learn about what the users expect. A step-by-step troubleshooting guide is also useful.


Vital for responding to issues, particularly with coded components such as Custom Search Commands & Modular Inputs. Be verbose enough to be able to remotely diagnose issues. Send your logs to Splunk’s logging directory and then your users can search over your logs in the “_internal” index via the Splunk Search App.

Listen to your users

They are a fantastic source of ideas and features. You’ll be able to think of many features, but the community will think of more once your App/Add-on is in the wild. Continually improve your App/Add-on based on feedback you get directly or via other forums such as Splunk Answers.

Support and Answers

Depending on the nature of your App/Add-on (community or paid for), your “SLA” will vary. I make my email available for direct contact and try to reply as timely as possible, but where possible I also try to direct users to Splunk Answers so that the community can benefit from the collaborative knowledgebase.


Some Apps/Add-ons might have so many potential use cases and target data sources that it is not really feasible to perform completely thorough tests. Leverage the power of the community to help you out.

Reference Architectures

Your App/Add-on might not just be deployed on a single monolithic Splunk instance. In all likelihood it will be deployed in a distributed Splunk architecture. Also, the target data sources will have their own specific architectural properties and deployment scenarios. So some simple reference guidelines for how to deploy your App/Add-on in various scenarios will be very useful to your end users.


Splunk Apps is good for distributing your releases, but is not a source code control system. I make all my code available on Github. In my mind this drives more community collaboration.

Legal Schmegal

OK, OK, I know you are a developer, but still be aware of any legal obligations i.e.: using 3rd party libraries or exporting crypto code.

Platform independence

I try to make all my Apps/Add-ons supported on all available Splunk OS architectures. This is just going to expand your potential user base. For example, with Modular Inputs, I prefer to code in Python and avoid using any native platform calls or libraries.

Choose the right language

Although Splunk ships with a Python runtime, one of the great things about the platform is that for the most part you are really not limited to any one language i.e.: you might use Java or C# to build a Modular Input if those languages were best suited to the task.


Depending on the nature of your App/Add-on (community or paid for), you should provide an appropriate license. When uploading an App or Add-on to Splunk Apps , several different open source license options are available to choose from.

Develop / Build / Test / Release Environment

A vital part of the equation. I use Eclipse with various plugins (Egit for Github integration, various language plugins), Ant for building my releases, Sublime for text editing, a few Linux and Windows Virtual Machines running on my Mac for deployment and testing. I’ve tried to make my environment as simple and streamlined as possible so I can turnaround releases in a timely and robust manner.

Social Media is your friend

Get the word out! Twitter, LinkedIn, Facebook etc. Github is also a great social platform for publicizing your App/Add-on.

Credit where credit is due

If you have collaborators/ contributors, share the love, be cool and acknowledge them.

Look and Feel

You don’t have to be graphic designer of the year, but it’s worth a little bit of effort in creating you own app icons and perhaps a custom CSS at the simplest. Perhaps you even have several offerings, so you can adopt a graphical theme across them all to tie in your “brand”. Be careful not to use copyright images though.


Be aware of naming conventions if you want your App/Add-on to be approved and published to Splunk Apps:


Aside from the actual technical aspects of creating Apps/Add-ons, those are the key learnings I’ve drawn from experience. Of course, over at there is a wealth of other technical detail about building apps, packaging and submitting them.


So that’s a lot of What, How and Where. In part 2 of this blog series I present some ideas on Why you should develop.

Damien Dallimore

Posted by