By Splunk September 27, 2017
Have you ever taken the time to think about your journey, be it personally or professionally? Have you ever said to yourself, "If I just knew then what I know now, things would be so much different."
Consider all the things that have shaped you to be the person you are today. The unnecessary risks you took, the chances you were given by others, the mistakes you made and learned from, the luck that you didn’t deserve but somehow had, or even the success or failures that took you in a decisively different direction. All of these experiences, for better or worse, shaped us in someway.
As a security practitioner, consider for a moment all that you have experienced over the years. What if you could step into a time machine? What knowledge would the “you of now” give to the “you of then”? If you happened to have this ability, what would the “you of now” need to know to be successful? But unfortunately, the Gray’s Sports Almanac has been obsolete for seventeen years, and we are now on our own to predict the next time the Cubs take the World Series.
I submit that being given the answer isn’t always the best answer. Rather, knowing how to ask the right question while discerning the situation, and where to make decisions surrounding the facts that you have within a given scenario, is a much more pragmatic and longer term solution for the modern security analyst.
The Whats, Hows and Whys?
I apologize in advance for writing a blog that requires one to apply the mental gymnastics of tearing at the fabric of space-time. However, just consider more generally as a security analyst, SOC manager, or CISO – knowing “the what” in terms of the risk you are researching or mitigating is sometimes easier than knowing the “the how”, or effectively communicating “the why” to leadership or other organizational stakeholders.
For example, as a security analyst, if I want to look for evidence of the DNS resolutions to evil domains so I can detect malicious command and control connections “the what” is trivial, but “the how” one actually goes about implementing the process (capturing, normalizing and ingesting DNS data), can become a bit more complex. Without additional resources, such as a recognized authority supporting the assertion that DNS monitoring is an industry best practice, data points highlighting the organizational requirement, effectively articulating “the why” to non-technical business leaders who cannot spell DNS and need to justify additional investment, can often be a catch 22 or violent collision of two worlds.
I will spare you the varying industry statistics that support the fact that there is a scarcity of security professionals within the market. I think you will all get it when I say that within a modern SOC, the day to day grit is dynamic, the challenges addressed yesterday are likely different than what needs to be dealt with today, and tomorrow will likely require some new hair on fire, hoop-jumping, highwire act that needs to be performed with limited time and resources.
So knowing all of this, how can we simplify the process by which Splunk customers can more effectively interrogate their security data? The assumption being, if customers know what questions to ask and are routinely enabled to ask the right questions, they can focus on interpreting the results, whereby they are more likely to make better decisions and take appropriate actions in a timely manner.
The Lost Art of Storytelling
As an analyst at heart, I have always enjoyed a good analogy to get a point across. For those of you who know me, my analogies are more miss than hit. So one thing I have been trying to do more of is tell stories.
Any of us can memorize and regurgitate details, however taking the time to understand all of the intricacies of a complex issue is another thing. By telling a story you can communicate an array of nuance and detail that can stay with the recipients long after your story ends. Your audience can also relate what you are telling them to their perspective and can eventually apply what they have learned when they encounter a similar issue. More importantly, they can also share these stories, helping others who might find themselves in a similar situation.
For example: It was 1996, I was just out of Army basic training and was acclimating to the desert heat of Fort Huachuca. It was the first day of class, where I peered around approximately two feet of various Army Intelligence field manuals stacked in front of me. I asked my instructor how I was going to memorize all of this in a few months. Where he shot back, “I have been doing this for 20 years, I have not memorized any of it, I just know where to look”.
Knowing Where To Look
While the rise of the machines hasn’t completely obliterated the need for analog books and traditional libraries, warehousing and organizing the bits that make up raw information or post processed knowledge is something the internet (and Splunk) does pretty well.
Running with that notion, you probably haven’t heard, but at .conf2017, Splunk is opening a Library. And I don’t really know how to put this…but it’s kind of a big deal. While there aren’t any leather-bound books, or smells of rich mahogany (perhaps in future releases), we will be unveiling a content subscription that we have dubbed Splunk Enterprise Security Content Update (ESCU).
This content subscription comes in the form of a free Splunk App and is primarily for our Enterprise Security users who would like to streamline the detection, investigation and contextualization of various security risks.
Within the ESCU Content Library we have given users a variety of features to explore and navigate subject matter within categories such as Vulnerabilities, Best Practices, and Malware. We have also included mappings to temporal analytic models such as the Kill Chain, so analysts can look for specific problems from across different data types (data models) to detect adversarial operations within the various phases of the Kill Chain. All of these categories can be filtered by analysts to best orient themselves toward a particular analytic approach, helping them find the best question to ask within a given problem set, and providing a perspective as to if retrospective analysis is required, or if a particular event is unfolding in the present.
The primary feature that we feel best encapsulates the degree of classifications and contextualization is within what we are calling an “Analytic Story.” In essence, an Analytic Story is a grouping function by which many classifications can be applied and where many different search types (we will explain more below) can be bound alongside a contextual analytic narrative.
Analytic Tradecraft and the Anatomy of an Analytic Story
Websters defines Tradecraft as “the techniques and procedures of espionage.” Now in this case we aren’t talking dead drops or grenade cufflinks, we are talking about sharing details of the analytic approach, where we disclose the techniques and procedures and the motivation behind bringing forth or illuminating the hidden truths within our security data.
I just want to take a moment to differentiate a key point here, and that is “Analytic Stories” are not indicators of compromise (IOC) bound as compared to other offerings. It’s not to say that IOC approaches are flawed or do not work, they certainly serve a purpose, and are quite complementary to what is offered within ESCU. At this time, we are simply focusing ESCU detections and investigative searches at the behavior level, or better known as tactics, techniques and procedures (TTPs). For more details surrounding the differentiation between behavior based (TTP) and IOC based detections I highly recommend reading David Bianco’s Pyramid of Pain.
Now, as we were. It is one thing to tell me how to detect something, but it's another to also give me the recipe by which I implement the detection capability within my own environment. The Research Team sought to make this process as transparent as possible so that others might be able to benefit and even improve upon what we have done.
So we asked, what is it that we need to share to make the process reproducible? What are the specific data types that customers will need to ask this question, and how might they go about ingesting the data, and which specific data models should they leverage?
All of the answers to these types of questions are included within an ESCU “Analytic Story.” In fact at the end of the day, an “Analytic Story” comes to life within a simple .JSON specification where fields can be added and removed as needed. In its current form there is a basic anatomy an analyst can expect to see when exploring an Analytic Story. It is important to note that Analytic Stories and the content within is dynamic, stories can expand and contract as we and the community identify new analytic methods and approaches and refine existing stories. In the current version of ESCU we have included fields for contextual narratives, various search types, their respective implementation details, summaries, as well as various categories.
Narrative: The Narrative will include an overview of what you are looking at with details as to how to detect and investigate, all the while conveying why it can be beneficial and what we are trying to achieve within the Analytic Story.
Analytic Story Searches: As we conceptualized the notion of an “Analytic Story” we considered the proverb “If you give a man a fish, he eats for a day. If you teach a man to fish, he eats for a lifetime.” So we recognized that just providing searches that allowed an analyst to detect something evil wasn’t enough, we wanted the next generation of analysts to know, and to understand what to do once they caught the fish. So we formulated the following search types.
Detection Searches: Consist of various behavior based searches that allow analysts to detect activity of interest. Often times there are more than one way to look for indications of a particular threat. Multiple detection searches can look for a particular problem from multiple angles (data types) to achieve a “zone coverage” effect.
Contextual Searches: Consist of searches which gather additional context for an analyst so that Enterprise Security Noteable Events can be evaluated holistically in the broader context of details such as, event X triggered and the user Y had logged in prior to the event triggering, and the following notables have also been triggered for that host.
Investigative Searches: Consists of the searches that help analysts dig into the results obtained from an initial Detection search. Effectively asking the secondary “follow up” questions that help analysts navigate the subsequent steps in the investigative process.
Support Searches: Consist of searches that fetch and or create data sets that directly support another Detection or Investigation search.
Searches will also contain specific details such as descriptions, instructions on how to implement them, as well as the likelihood of false positives and conditional scenarios by which they might manifest. We will also include an ELI5 (Explain it Like I am 5) section which seeks to simplify the Splunk Processing Language (SPL) and boils it down into a human readable format for the aspiring Splunk ninja. (You’re welcome, security leaders.)
Analytic Story Mappings: We have included various mappings to industry recognized standards, analyst models, and best practices such as the Center for Internet Security Critical Security Controls, MITRE ATT&CK and the Kill Chain. We have also highlighted the specific Splunk data models and example technologies that can be used to collect and normalize the requisite data sets needed to identify suspicious behavior and answer the associated questions. Finally, we have included references and citations so that analysts who need more background context as to what might have inspired the “Analytic Story” can read up and find additional details.
Automation & Adaptive Response: Analysts are busy. We get it, we have walked many a mile in your shoes. So we have begun to explore where we can introduce orchestration and automation within the “Analytic Story.” We have included a custom Splunk command “Run Story” which executes all of the Detection searches on behalf of the analyst. We have also included Adaptive Response capabilities so Splunk Enterprise Security users can configure ESCU contextual or investigative searches to execute as a follow on Adaptive Response Action once a Detection Search yields a Notable Event of interest.
Don’t Leave Me Hanging
Obviously for those of you who are familiar with the intelligence cycle, evaluation and feedback are critical to the overall process and permeates each phase. So we have included a Feedback Center which will allow ESCU users to engage with the Splunk Research Team directly.
Just like any storyteller or your favorite 80s cover band, we are taking requests. So if there are any general issues, specific questions, or even kudos and words of encouragement that you want to pass along, please send us a note. The only way we can improve is to gather feedback and hear from our users.
Conclusion
If there is one key point to remember, it is that there is a difference between knowing something and fundamentally understanding it. Splunk’s Enterprise Security Content Update and our “Analytic Stories” seek to go beyond just knowing, by giving analysts and their leaders what they need to understand the bigger picture how you plan, collect, process and analyze data associated with a given risk.
Our goal with ESCU to carry a bit of the load for everyone. If you are a security analyst and Splunk Enterprise Security user, we want ESCU to give you a running start on your day to day workload; perhaps there are a few tasks we can take off your plate in terms of how you go about hunting for a particular threat. While you don’t have to be a Splunk ninja to get started, over time and through osmosis, muscle memory can take over to the point where you can pick and choose what you need to construct your own specific security searches.
As SOC manager, you want to increase efficiencies across your team, so perhaps ESCU can begin to free your lesser experienced analysts to operate more independently whereby allowing your senior analysts to work on higher priority items, building your bench of up-and-comers.
Alternatively, when it comes time for budgeting, ESCU can help you defend the value of your supporting investments (such as endpoint collections), highlighting the detections (wins) where you had data and could take an action, versus navigating through a blind spot (intelligence gap) within your security data.
If you are a CISO, you just went to the mat and secured a hefty investment, so the stakes are high for success. You want to ensure that your people and tech investments are optimized and working together. You want to see your Enterprise Security investment and your analysts coming in line and bearing fruit as soon as possible. ESCU supports that connective tissue back into Splunk HQ that seeks to erode the time to value and get your security engine firing on all cylinders.
If you would like to to learn more about ESCU, check out the Splunk Enterprise Security Content Update documentation, or you can download Splunk Enterprise Security Content Update directly from SplunkBase.
Rich Barger
Director, Security Research at Splunk
@richbarger
Follow all the conversations coming out of #splunkconf17!
----------------------------------------------------
Thanks!
Rich Barger