Splunk for Security Investigation: Threat Detection

Welcome to the Splunk for Security Investigation Experience. In this video, we look at authentication failures as a mechanism for investigating security issues.


Video Transcript

Splunk enables rapid security investigation and analysis. We will be walking you through a quick Splunk security investigation experience to show you how Splunk can immediately help you identify indication of compromised and quickly determine the scope and the cause of threats.

In this exercise, we look for patterns of authentication failure across our entire infrastructure to detect potential bad actors. The result you see on the screen can be found using the SPL search command above. We will be looking for any event with failed password in them. As we search fail, Splunk's search ahead feature will return variations of the search that match fail. This can help us to refine what to look for.

By using a wild card or star, we'll get results for any event that begin with fail, including fails, failed, and failures. For password, notice the search ahead feature shows there are events where password is equal to specific values. We can select one of these pre-populated searches to find the events that match specific criteria or just events that match password.

So far, we can see a simple fail star password search found all failed system access attempts across our entire infrastructure according to our event logs. And with search ahead, Splunk can help guide you through the investigation process. Our result shows there are approximately 2,500 events that match fail star password.

The timeline panel shows the distribution of matching events over time. Using the histogram, you can zoom in and out of frames to understand distribution of events over time. Here, you can select the time range. These are the fields associated with the resulting events and the raw events with the matching terms highlighted.

To determine what systems are affected, we look at the type of events. This is represented in Splunk as a source type. Source type indicates what type of data it is. So in this example, we see four source types that contain the search pattern fail password, which represent Windows, Linux, database, and a file server. You can track down authentication failures in Splunk with a single search versus querying from four different tools or data sources.

Splunk allows you to find a pattern across an entire technology stack regardless of the type or source of data. Now that we've narrowed our initial search, we can look at fields extracted from the raw data to continue our investigation. First, we'll select the fields necessary to analyze the failed authentication attempts.

We went to select destination, that shows where they are trying to go, source, that shows where they originated from, and the users that are associated with failed logins. We'll select these from the fields panel in the left and move them into the selected fields panel. Starting with the dest field, this indicates servers or hosts that are being accessed. We can see this represents 60 different hosts, and this shows all the top target hosts that someone is attempting to access.

E-commerce-03 has more than 1,400 log login failures, and AD-19 server has several hundred access failures. And we see authentication failures on several other hosts. This shows us where the majority of our attacks are targeted. The source fields indicate what workstations are originating the most login failures, and where they are located. This host seems to have some usual amount of activities, and POS terminal is also contributing to the high number of authentication failures.

The user fields indicate which target users have the highest number of login failures. Browsing the fields in the exploration panel provides fast context into where the account failure attempts are originating and the targeted accounts or assets. Now we've got the information we need to continue our investigation to make linkages between which hosts are attempting to log in to which target hosts using which accounts. So we'll use a pipe operator and then the stats command to help us aggregate the number of failures by origin of access, target system, users on the target system, and the type of system.

We see the stats command calculated the total number of authentication failures associating the failure counts to the origin IP, target host, and user credentials used to access the system. Next, we can use the sort command to show descending order and highlight the highest number of login failures by session. Session being a combination of the unique source, destination, and user.

Finally, we apply the Where command to show only authentication failure activities by a source client to a host with more than two failure attempts. The ability to apply conditions on calculated results, in this case, sum of total failed log and attempts, makes Splunk a powerful security analytics platform. And we can do some more advanced search examples to demonstrate the possibilities.

For now though using Splunk, we've discovered potential threat activities by finding authentication failures from across our entire network. And then we've applied calculations and logic, so we know where to focus our investigation. We've got two issues here. First, someone has tried to access the e-commerce-03 host more than 1,400 times. That's most likely a scripted attack.

But there's also this host with the IP that seems to be probing the network by accessing multiple web servers in the cloud, and more critically, attempting to gain access to this database-001 server. Through this demonstration, we have shown how Splunk can quickly search and detect potential threats using simple Splunk search commands or SPL. In the next video segment, we'll drill in to further validate the threat activities we've found. Stay tuned.