Learnings From a Scripter Turned Splunk App Developer

So, you have a burning desire to develop apps on Splunk? Easy! Just start saving searches, throwing them at dashboards, and send it to Splunk for approval. Before you know it, you’re a published developer on right? – Probably.

I recently wrote the Security Monitoring for Splunk app and faced a whole lot of lessons. Mostly those lessons were learnt by trial and error, but as with all good things at Splunk, we stand on the shoulders of giants and I had no shortage of great advice from others too!

In case you haven’t checked out Security Monitoring for Splunk, it is a single aggregated place to monitor and visualise patterns and trends across all of your security feeds in a single app. From system logins to firewall and web proxy traffic, IDS alerts and lots more - it's all there. If Security Operations is your ‘thing’ then I highly encourage you to check it out!

Anyway, back to the point. I thought I would write up my experiences and share them with the community while fresh in mind. My hope is this blog might help others on a similar voyage. Before I go on, a quick disclaimer; I am not the final say in the world of writing apps on Splunk. I’ve done my fair share of development over the years (Splunk and otherwise), and understand that the customer journey is king, but that by no means makes me an expert. I’m just a hacker hacking his way through a day’s work by any means necessary! Now that’s out the way, here are some of my lessons:

Planning and Preparation Prevent Poor Performance

Start with a design goal in mind, and get it typed up before anything else happens. What should your app achieve? What will it not achieve by design? After what could be months and hundreds of hours work, you’ll be thankful you did. You’ll go back to it time and time again for guidance when you go off piste, or lose your way on the original design objectives.

Flexible But Not Compromised

Use your typed up design document as you go through the development process. It’s most likely you’ll have further great thoughts and additions as you progress with writing your app. Tweaking your original goals and objectives are healthy if done consciously. Revisiting your original thought process means you can let it evolve, without compromising your overarching goal because you lost sight of the original concepts.

UX Design and Colours are Not Just “A Thing”

They are an art form! I spent a lot of time trying to get my user interface to flow, look pleasing on the eye, and not have too many clicks to get to an answer. I’ve read web pages on design, took the advice of esteemed colleagues, flailed around aimlessly with themes of colour - and whilst I got to a point of ‘good enough’, I’m still convinced there’s much improvement to be made. Huge respect to those who can design! I came across this image on for working with colours, which I found helpful:

Know Your Audience

Every developer wants their creation to be used far and wide. Understanding the users of your app, what they need from it, and how they will use it are key to your hard hours of coding. Placing the needs of the user over fancy or complex design is what will result in the app being used for its intended purpose.

Perfection is the Enemy of Good

Grand designs, fully functional and heavy feature sets are always the creators dream - but what good are dreams if they never become a reality? Know when to let your creation set sail, and bring the updates that users think are important to them over a period of time.

Realism and Pragmatism

Once you’ve let version 1.0 loose, be cognizant about what time and energy you’ll be prepared to expend on its evolution. Maybe your app meets all its criteria and won’t need to evolve. Maybe it’s a fully implemented API, but what happens if the vendor updates that interface? Will your app evolve too, or will it expire its useful life and die a slow and painful death? My view is once you have created something, you must then help keep it relevant for the good of the community.

I know there are lots more lessons for me to learn personally about developing apps on Splunk, and if you’ve got great advice and ideas on either Security Monitoring for Splunk, or your experiences of apps on Splunk - I’d love to hear them!


Derek King
Posted by

Derek King

I've had a long and meandering journey to Splunk, with (ahem) 20 years in technical roles from application development, OS engineering, Networking, and the last 10 years fell in love with all things cybersecurity. At Splunk I help customers out in any way I can, from understanding the basics, to doing cool cyber stuff with it!