A Salacious Soliloquy on Sysmon


This blog post is part sixteen of the "Hunting with Splunk: The Basics" series. Splunk's security team is addicted to using Sysmon for endpoint data. Sysmon is so much fun to use it almost makes me want to go back into operational security...almost. John Stoner writes an excellent introduction to Sysmon as a data source in Splunk and how to use it. I hope you enjoy! – Ryan Kovar

If you have been reading our hunting series, you may have noticed that many of the hunting techniques we have talked about have centered on network-centric data sources. While there have been a few host-centric data sources mentioned, we have yet to speak about the big kahuna in our hunting tool chest. We are going to rectify that today because we are going to talk about Microsoft Sysmon!

Sysmon is a detection technology; it's not for prevention. Many other products perform blocking/prevention, but if we need insight into what's happening, Sysmon provides an excellent, cost-effective method.

Microsoft Sysmon has been around since 2014 and can be found on the Sysinternals site. Mark Russinovich and the Sysinternals team had built many great Windows utilities and tools, and Sysmon is a continuation of that since their acquisition by Microsoft. Sysmon provides more in-depth insight into what's happening on a Microsoft system and can tell you when processes are starting and ending, what parent command started the process and much more! Beyond that one killer feature, it can report network connections from a host and many other system states that provide greater insight than just using Windows Event logs.

There has been plenty written about how to configure Sysmon so we won’t cover that (besides, we're here to talk about hunting), but let’s talk about the elephant in the room. We know what you're thinking—you want me to log everything my workstations do? We're not suggesting that, but we are suggesting that with a little bit of tuning you can get the essential nuggets out of Sysmon. The first thing you need is to configure a monitoring template to determine what will be collected. There are a number of these templates out there already.  In fact, we like the SwiftOnSecurity Sysmon template available on GitHub to jump start our configuration.

"That’s great," you might say, "You give us a template, but I am still concerned about generating too many events—what now?" Well, we'll also point you toward the work that TransAlta did. TransAlta presented last year at .conf2017 and highlighted how they were able to filter down Sysmon to about 10MB per day per workstation and still gain actionable information. With this background out of the way, let’s dive into Sysmon.

Let’s start by taking a look at the details found in Sysmon. In our example below, we can see that the executable 3791.exe is being executed from the directory c:\inetpub\wwwroot\joomla. The EventDescription of Process Create is one of many kinds of events collected by Sysmon, but the process creations alone can be incredibly useful when hunting. As we continue to look through the event, we notice a field called ParentCommandLine. This field contains the value cmd.exe /c "3791.exe 2>&1" which was parent process of 3791.exe. Additional essential pieces of information that we can gather as part of this process creation event are the hashes; MD5, SHA1, SHA256, and IMPHASH. Finally, key information including the host the event was collected from, the directory, the process and parent process ID are also collected.

Now that we have a background in the data found in Sysmon let's apply that to a hunt.

During our hunt, we have identified the file, 121214.tmp, on a workstation. We need to understand more about this file and what relationships it might have with other processes. Let’s start with a simple search like this:

index=botsv1 sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational CommandLine=*121214.tmp* 
| table CommandLine

Here we can see all of the instances where 121214.tmp showed up in the command line. These instances provide some interesting information including that this process was killed or that cmd.exe runs and then triggers 121214.tmp to run.

When we hunt, we likely will want more context than just what executed. We could craft a search that can gather what was executed and associate it with its parent process to understand what process executed and then what preceded and followed it.

index=botsv1 sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational 121214.tmp CommandLine=* 
 | table _time CommandLine ProcessId ParentProcessId ParentCommandLine 
 | reverse

In this case, we can track our file by looking at the ProcessId and ParentProcessId and their associated CommandLine and ParentCommandLine values and chain them together. We can also take this a step further and continue searching for other ProcessIds that match our ParentProcessId to see their relationships.

With our search, we see a series of processes that executed, concluding with 121214.tmp terminated by the taskkill command and a ping of the loopback address occurring. The other thing we can see is that the ParentCommandLine where the 121214.tmp was first seen is a wscript.exe that calls the file 20429.vbs from Bob Smith’s roaming profile directory.  Wscript.exe is a legitimate Windows application, that is used to run VBScript files.

Fortunately, because we are logging this with Sysmon, we can easily modify our search and look for CommandLine and ParentCommandLine when either the parent process or the process ID is 3968.

index=botsv1 sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational (ProcessId=3968 OR ParentProcessId=3968) CommandLine=*
| table _time CommandLine ProcessId ParentProcessId ParentCommandLine
| reverse

Now we can see what appears to be VBScript executed directly from cmd.exe. We could continue iterating further and further back if we chose to as well if that was required in our hunt.

The last thing we will mention in this blog is around the operationalization of a hunt. Ideally, we should not be hunting for the same things over and over, so if we find something that's of value, we should operationalize it and alert the incident response team. Using our previous example, we identified what could safely be referred to as an exceptionally long string in the CommandLine, and we want to alert in the future these kinds of events. Using the eval command we discussed earlier in this series, we could build a search like this.

index=botsv1 sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational CommandLine=*
| eval lenCL=len(CommandLine)
| where lenCL>1000
| table _time CommandLine ProcessId ParentProcessId ParentCommandLine lenCL
| sort - lenCL

In this search, we calculated the length of the CommandLine field and then filtered on lengths of CommandLine that were greater than 1000. If we wanted to use a calculation other than a fixed value, we could use the stats command to calculate a standard deviation instead. Any event that returns could be sent for investigation and action by the incident response team.

We have only scratched the surface of what Microsoft Sysmon can do, but we hope that this was enough for you to go check it out, install it and use it as part of your hunt. Sysmon is a valuable addition to your arsenal, and by gathering these events, it opens up your world to greater insight into what your Windows systems are doing!

Happy Hunting! :-)

John Stoner
Posted by John Stoner

I grew up in Virginia, graduated from Penn State and came back to Virginia where I have worked with databases and cyber for over 20 years. A job or two felt like I really was a cast member of The Office or Office Space, but every one of them taught me something new.

Join the Discussion