Spotting the Signs of Lateral Movement


This blog post is part twenty-two of the "Hunting with Splunk: The Basics" series. Lateral movement is one of the key indicators when you ACTUALLY have an APT in your network. Finding it can be difficult because adversaries often use legitimate credentials to move around your network. Derek drops some sweet Splunk knowledge below on how to use Splunk to detect those baddies in your network. – Ryan Kovar

If you've been keeping up with our "Hunting with Splunk" blog series, you’ll know we shifted focus from network-centric data to the big kahuna of endpoint data, Microsoft Sysmon. If you missed it, then you have work to do my friend! My advice—go grab a coffee and buckle up for two blogs all in one!

Once badness makes an inroad into your network, the adversary has a set of goals—steal credentials, persist, find the good stuff, exfiltrate the good stuff and get paid! To do that, they need to move laterally either by using exploits against other vulnerable hosts, or by using legitimate tools but for malicious purposes. In this post, we’ll focus on using legitimate tools for badness.

First things first, if you’re not capturing Windows event logs from your endpoints, you're going to really struggle with hunting for and detecting lateral movement. It’s like fighting the English fog with one hand behind your back and an eye patch! Fortunately, I don’t need to roll out on my high horse on the matter of endpoint logs since our very own James Brodsky has been cantering around on his for some time at both .conf2015 and .conf2016.

Our hypothesis is that legitimate Windows tools can be used against us for moving laterally within our network. How might the adversary be hopping from one machine to another without exploiting vulnerabilities? Some long-established tactics are well known; remotely creating WMI processes, scheduling tasks and creating services are often seen.

Psexec is a great sysadmin tool that allows administrators to remotely connect to other machines and carry out admin tasks, and it's often found (legitimately) on networks. But what if psexec was used to gain a remote shell or execute a PowerShell cradle on the remote machine? Let's look at how we can hunt for this type of activity. When looking for lateral movement, we're identifying processes connecting remotely into a host. Our initial search could use Windows security logs, looking for authentication events over the network from rare or unusual hosts or users.

index=wineventlog sourcetype=WinEventLog:Security (EventCode=4624 OR EventCode=4672) Logon_Type=3 NOT user="*$" NOT user="ANONYMOUS LOGON" 
| stats  count  BY dest src_ip dest_nt_domain user EventCode 
| sort count

The EventCode for a successful Windows logon is 4624, the LogonType of 3 is a network connection and 4672 privilege escalation events. To keep our search tight, we exclude computer logons (*$) and unauthenticated sessions (ANONYMOUS LOGON). With our result set, we count the events for each host and display the source IP address, EventCode and user that authenticated. Finally, we sort them in ascending order to surface rare events.

Using this information, we begin to lean in on hosts of interest. In particular, we see that administrator has logged into host Win7-2 from IP address, which we're not expecting to see. This is definitely not an admin or IT support address; this is another client machine on the same subnet. Interesting and warrants a closer inspection!

We can use Splunk to drill directly into these raw logs by clicking on the row and viewing the events.

We now see the time the events occurred, two of which happened in fairly close proximity to one another. We can now modify our search to query all events from that host and narrow the time range to focus on activity occurring around these two events to see what else happened on that host.

index=* sourcetype=* host=win7-2

Immediately, we see entries for service creation (EventCode 7036) for psexec from the WinEventLog:System log. At this point, we know the administrator connected remotely and ran psexec, but we have no idea of the context, good or bad! What did psexec actually run? Fortunately, if you're logging process creation events, we can answer that question! In our environment we collect Microsoft Sysmon logs (and I know you know that we love Sysmon!).

We could run another search for process creations, but our previous search returned a number of events with the same timestamp. This warrants additional investigation.

If we expand the first Sysmon event by clicking the right chevron (>) next to the event, we can see the psexec service executed cmd.exe.

Based on our searches, we now understand the user administrator connected over the network from and gained command line access to our victim host. If you’ve already read “A Salacious Soliloquy on Sysmon,” you'll likely be rushing off to operationalize your searches.

Using psexec for lateral movement has been around for quite a while and is still very popular and relevant. But wait! There’s more… A new kid is on the block and just like psexec, it's a legitimate tool being employed for badness!

Distributed Component Object Model (DCOM) used with Direct Data Exchange (DDE) allows an adversary to traverse the network using built-in tools. If you are not aware of DCOM, it’s an extension to COM that provides a client / server architecture, allowing application communication across a network. If you're not aware of the DDE protocol, it's the process that enables data sharing and reuse between Microsoft Office applications.

Let’s take a look at how our previous approach stacks up against this technique.

Running the same search looking for network logons produces the same results, as we would expect since any connection needs to be authenticated.

index=wineventlog sourcetype=WinEventLog:Security (EventCode=4624 OR EventCode=4672) Logon_Type=3 NOT user="*$" NOT user="ANONYMOUS LOGON" 
| stats  count  BY dest src_ip dest_nt_domain user EventCode 
| sort count

Following exactly the same process of broadening our search for all events on the host and narrowing the timeline shows no evidence of service creation, but again Sysmon comes to our aid with process creation events.

index=* sourcetype=* host=win7-2

We see a parent process of Excel.exe and the command line executing calc.exe (not normal behavior!). If you have a keen eye, you’ll also spot the ParentCommandLine using “/automation -Embedding” which occurs as a by-product of DDE creating the Excel object. Using this approach, we can hunt for one of the latest lateral movement techniques (DCOM), as well as one of the more established (psexec). Great!

From here, we should operationalize the searches and look for remote logins, service creations, and processes with “/automation -Embedding” in the ParentCommandLine in addition to parent processes that don’t look normal using the techniques outlined in “A Salacious Soliloquy on Sysmon”. Oh, and remember, filtering by time is always your friend!

We have touched on two different ways in which an adversary can traverse the network and we did this with only three sources of data—Windows Security, System events, and Sysmon. Other data sources like network metadata and registry entries can also be used for spotting lateral movement. If you would like further inspiration, take a look at the MITRE ATT&CK framework.

Happy Hunting!

Derek King
Posted by

Derek King

I've had a long and meandering journey to Splunk, with (ahem) 20 years in technical roles from application development, OS engineering, Networking, and the last 10 years fell in love with all things cybersecurity. At Splunk I help customers out in any way I can, from understanding the basics, to doing cool cyber stuff with it!

Join the Discussion