Community Spotlight: Why Will Searle Swears by $tstats$, Git, and the Forums

For this month’s spotlight, we’re sitting down with a long-time member, Will Searle, who embodies the ultimate "every day is a school day" mindset. From accidental beginnings in 2013 to mastering complex Splunk Cloud migrations and Git-managed ES rules, he’s done it all. When he’s not busy automating admin tasks or dropping knowledge in Splunk Answers, you can find him gearing up for his sessions at .conf26 in Denver or bumping everything from Linkin Park to Ibiza classics. Let’s dive into his journey and his top $tstats$ tips.

____________________

Your Work

What’s the most satisfying part of your 'day in the life' as a Splunk Consultant?

Every day is a school day for me, when I login I don't necessarily know what the day has in store for me or what challenges I’ll face. What ultimately satisfies me the most is being able to overcome any of the challenges or hurdles that the day brings, and it's not unusual for a bit of learning along the way!

How did you get involved with this work?

Back in 2013 whilst working as a Web Developer I applied for a job that involved working with ‘Splunk’ (which I hadn't heard of!). After some research, I felt a little more confident about what it did and managed to convince an interview panel that I’d be a good candidate for the role! The role was a good mix of Splunk operational work along with platform infrastructure / Site Reliability work which led me to take on a role in a different team managing the infrastructure for a national healthcare system. Whilst this was a fun time with its own challenges (the birth of ‘the cloud’) I ultimately gravitated back to the Splunk world but with more focus, migrating the many disparate deployments across the organization onto Splunk Cloud. Then Covid hit and things took a twist; we needed ITOps monitoring like we’d never had before. We leveraged our existing ITSI deployment on Splunk Cloud and quickly began monitoring data flows and IT systems relating to the testing, tracing and vaccinations of Covid—at this point I’d seen so much of the Splunk product and ecosystem that I began to weigh up my next move. In November 2020 I left the safety-net of my permanent role to go contracting and start delivering Splunk Professional Services work, which I’ve been doing since! Working with customers across the UK to help solve their biggest challenges and share my experiences (good and bad)!

What has fed your interest in your work?

I’ve been using computers from a young age, starting with Macromedia Dreamweaver at the age of about 11 to build websites (which didn't really get many visitors!) With that evolving into becoming a Web Developer after college, I've always had a strong affiliation and interest in web/app/system development. When I moved to working with Splunk I fell into the App Development side of Splunk quite easily and loved the fact that there was a platform that I could just build apps on top of without having to ‘spin up’ a new environment somewhere. The ecosystem and how I was able to get data from anywhere and visualize it in one place was something I kept coming back to. Since the early days, I’ve been interested in pushing the boundaries of the Splunk platform and finding the niche development, operational and deployment techniques which I’ve found actually help solve the most challenging use cases. Ultimately, I think the versatility of the platform has kept me interested in continuing to work with it, but in recent years has allowed me to share with the wider community.

____________________

Working with Splunk

How do you use Splunk in your role? For example, are there any interesting use cases you’ve experienced in using Splunk

From one day to another I have different hats, which are great for me. I move from building apps to architecture work, designing dashboards, and troubleshooting searches and deployments. Some of the most interesting things I’ve done recently are managing ES rules (and wider ecosystem) from Git, as a source of truth. This approach means that rules can be developed in one environment and pushed to production through a carefully managed peer-review and approval process to ensure that no mistakes are made and that there is parity between environments and code control. Nothing makes it into production without an additional second pair of eyes going over it—reducing risk and improving transparency and audit.

What was the specific problem you were trying to solve when you first found the Splunk Community?

I think one of the first things I came to the community for was in search of Icons to build a topology diagram, ultimately it didn't exist at the time and I ended up answering my own question with a link to the icons I’d managed to get hold of—I still refer to that thread regularly and there are also a couple of other questions I’ve asked over the years which I can never remember the answer to and therefore keep finding my way back to the old question I posted on Community.

Can you tell us about a positive experience you’ve had with the community?

There are so many people who share their experiences on the community and they’re all so valuable—sometimes there is more than one ‘right’ answer to a question, sometimes there are nuances to the users environment which mean that one approach would be superior to another, even now as a frequent contributor I find that others are able to challenge (rightfully) my answers and provide alternative points of view, meaning that the users asking questions are getting a combined response from many years of experience! That's value you’d be hard pushed to get from anywhere else!

What are your top 2 Splunk hot tips?

1. Use tstats: I've done a talk about this to colleagues in the past and often find that searches modified to use tstats run over 10x faster. There are so many cool examples of tstats searches and a Conf2020 talk (PLA1089C) which I’d recommend reading.
2. Automate things: Humans make mistakes; we’ve all done it! If you can script/code/automate part of your admin tasks, then you might not give that person access to that PII data by mistake because it’ll get picked up in peer-review before hitting production 😉

If Splunk AI were a teammate, what role would it play on your team?

I think that Splunk AI can help fulfill a number of roles within a team, as a product/tool its fast evolving and can do much more now than this time a year ago. I think its like having an assistant that can help pull together information quickly whilst you’re focusing on something else, which means you save time and get more done.

What’s one problem you hope Splunk AI will help solve next?

I’d like to see more automated alert actions / next steps. In other words, I want it to know what I would do/search for when receiving an alert and provide the next set of data to me without having to do it myself. This means as soon as I pick something up, I would already have a wider picture over what is going on.

____________________

Keeping up With the Times

What’s your best way of keeping up with industry news?

I download the latest version of Splunk when released and scour the release notes and conf spec files to see what has changed! It's important to me that I know what has been released and how my customers can leverage it. I also sign up to programs in the Voice of the Customer (VoC) portal so that I get an idea of product roadmaps and where things are going. In terms of the wider eco-system, I do a fair amount of reading online about where things are going.

Is there anything you’d like to shout out or elevate?

I’m doing several talks at .conf26 in Denver and would love to see packed out rooms so please keep an eye out for the session catalog and come and find me if you have any questions!

____________________

Life After Hours

What are you watching or reading right now?

I have a backlog of training material to watch but I’m also reading a book to improve my public speaking skills. Let's see if it helps! Fingers crossed.

Who’s your favorite musical artist or band?

I have a wide range of music tastes; I can listen to an Orchestra play Ibiza classics one hour and Rock the next. The last two concerts I went to were both to see Linkin Park (with the new singer, Emily), one as a stadium tour but one at the O2 arena in London—the performance was amazing, probably the best I've ever seen because the sound was so immersive and I was stood so close. I saw Eminem in 2011 (with D12) but really hope he does another tour soon as would be worth seeing again!

____________________

Insights to Share

What advice would you give someone who is up and coming in your field of work?

Try and get experience tackling a variety of problems, like we see on Splunk Answers. This allows you to quickly adapt to the situation and work through the problem methodically and efficiently. Working with Splunk often isn't just about the Splunk products, but also the wider ecosystem: Linux/Windows/Networking/Data Science/Python/Gitlab/Azure/AWS etc are all good examples of other skills worth investing in.

Khoros is the 'Library of Alexandria' for Splunk. Is there a specific solution or documentation you authored that you’re most proud of seeing other users reference over the years?

I authored a post recently after a head-scratching issue with a customer of mine and wanted to make sure others didn't waste their time investigating it too...it seems like it was a wider problem than I first thought and I was pleased to get over 30 karma for a single thread!

The other was my first post, about Icons—I still find others referencing it to this day, sometimes it's simple things which aren't easily accessible that hold people up!

When you’re diving into a complex 'unanswered' thread on the forums, what is your personal process for deconstructing a problem that everyone else is stuck on?

If it's unanswered, it's often either a niche problem, a lesser-known part of a Splunk product or the question isn't well written. If it's the latter, then a simple reply to get clarification from the author is the first thing to do. If it's a lesser-known part of the product then it gives me an opportunity to read/learn about it and try and work through it and if the problem is niche then really the best way to try and address it is to put myself in the author’s shoes and try and work through it, using previous experiences and edge-case approaches until you reach a solution which doesn't break anything and delivers! Sometimes these approaches can be unusual, but I think a detailed explanation covering the risks/caveats etc. would typically give the author enough to make an informed decision on how to move forward.

____________________

Want To Be Part of the Community?

The best way to get better at Splunk is to hang out with people who are doing it at the highest level. Whether you’re looking to solve a 'white whale' query on Khoros, want to talk shop in real-time on Slack, or are looking for your local crew in a User Group, there’s a seat for you at the table.

Join the Splunk Community