By Ryan Kovar October 09, 2019
We’ve had many requests from Boss of the SOC (BOTS) contestants asking how they can “up their game” and finish in the top ten. I figured the easiest way to find out was to ask someone who has! With that in mind, I spent some time recently talking to my old friend Brad Lindow, a Splunk security expert at ServiceNow who was 25% of team “Blue Vulture1” that finished third at the gigantic Splunk .conf18 Splunk BOTS competition!
So Brad, how did you and your team prepare for BOTS?
The first thing we did was start researching previous BOTS and reviewing Splunk Blogs for anything that the BOTS admin team might release before the event. Then we downloaded the open-sourced data sets, loaded it up, and started practicing.
That's a great idea since we release a massive amount of content and it all feeds into or from BOTS! We expressly publish articles about new data sources or methodologies in our "Hunting with Splunk: The Basics” series! I’m curious, how did you “action” that information that you gathered in the heat of the moment?
We actually used Splunk to organize the info. Using the open-sourced datasets, we started building dashboards and prepopulated generic searches. Then we would add explanations for each search and data type and hyperlink the relevant blogs or articles that we got the information from. Then when we started the competition, we copied over the dashboards to our search servers and off we went!
I remember you telling me this during the competition, and I was blown away. I honestly think y'all did more pre-work than any competitor I've ever talked to. Do you have any specific examples of searches or dashboards that you used last year?
Sure thing! First, we looked at the “Spotting the Adversary” blog post from a couple of years ago. It's a bit old but still very relevant. The other blog post that was immediately useful talked about lateral movement.
Oh wow, I’m curious what you found in the “Spotting the Adversary” blog?
Well, that's just an example. We actually found every blog published by your team and extracted any/all info that was useful (like Windows event codes) and put them in a dashboard. Another example would be event codes that Michael Gough talks about on his website. Specifically, the 6 event IDs talk that he gave at .conf a couple of years ago.
I also noticed that your team was dead silent. How did your team communicate and stay on task during the event?
Easy – Slack. We created our own group and talked there instead of giving our answers away by talking. This also allowed us to share info and searches really quickly. We also tried to organize by each team member being assigned specific questions, but that didn't work out perfectly because some questions built off of each other. This meant that by the end of the competition, we were occasionally all working on the same question rather than being efficient and working off on different tasks.
Oof. Yeah. For the harder questions, we do often build off of previous easier questions, so dividing them up by just "question" number doesn't always work. However, this year I'd recommend splitting up by "scenario" rather than number since we don't have questions in different scenarios build off each other.
We thought about that, it's just in the heat of the moment you do what works. For us, it was just banging away until we were almost on top of the leader board.
I'm curious, some of the questions required the use of Splunk products like Splunk Enterprise Security and Splunk Phantom. Did you have experts on your team for each of those tools, or did you just "figure it out"?
Honestly, this was kind of hard. I was the only person on the team that had been exposed and used Splunk Enterprise Security and Splunk User Behavior Analytics. Luckily, most of the product questions had really good hints and seemed like they were designed to walk you through the tool rather than make you suffer for not knowing the right answer.
Yup! You figured out our crafty goal. The Splunk product questions are designed to introduce you to the tools but are not required to have knowledge of them to answer. However, if you DO know the tools, you can solve almost all of the other questions faster and easier.
We just dedicated one person to work on the 100 series (these were the tool questions) and then spread out on the rest of the questions.
Speaking of questions, a final one for you: Did participating in BOTS help you and your team after the competition?
Totally. It made us more excited about Splunk and its different use-cases we hadn’t thought of. Also for us, it’s always better to learn by doing. So having Boss of the SOC as an activity to work together on as a team with a purpose was much better than having to sit through training. It really is a unique activity.
Big thanks to Brad, his team, and the 719 OTHER people who “officially” played BOTS last year at .conf (yes, we caught you for those who know what we are talking about). I hope to see you all at .conf19 this year where we host the biggest BOTS ever!
Follow all the conversations coming out of #splunkconf19!