CloudTrail - Digital Breadcrumbs for AWS

This blog post is part twenty-two of the "Hunting with Splunk: The Basics" series. We've talked quite a bit about Microsoft Cloud in this series, now it's time to talk about the big Bezos in the room: AWS. Matt Valites is new to Splunk and brings a wealth of awesome experiance around hunting in an AWS world. I hope you enjoy! – Ryan Kovar

As part of this year's Boss of the SOC (BOTS) competition, the Security Specialists incorporated an AWS web-hosting infrastructure into the game's scenario. Like many of our customers, Frothly—the game's home brewing protagonist company—migrated some of their infrastructure to the cloud. This gave us an opportunity to play with real-world data to answer many of the questions we hear Splunk customers asking:

• What data is security relevant?
• Where and how do they get that data?
• Moreover, most importantly, how can that data be used to detect malicious activity?

In this blog post, we'll explore the most pervasively useful data from AWS: CloudTrail. At a high-level CloudTrail data provides digital breadcrumbs of activity in your account, so you can both piece together what happened and deploy proactive measures of detection.

What Is It?

Amazon Web Services (AWS) defines CloudTrail as "a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account." Any user, role, or service that attempts successfully or unsuccessfully to act as a service in AWS will generate a log containing information about that event. CloudTrail is such a valuable data source for security operations that AWS now enables it in your account by default.

AWS services are the products that Amazon delivers. Compute via EC2, Relational DataBases via RDS, and even logging in CloudTrail itself are all examples of AWS Services. By enabling CloudTrail, users get details about all of the services currently offered, as well as any new products that AWS will release in the future (and that your users will inevitably use in some capacity).

Getting It In

CloudTrail can be ingested into Splunk via the Splunk Add-on for AWS using a number of methods:

S = Supported, R = Recommended

In their environment, Frothly configured Splunk to ingest CloudTrail data via the recommended SQS-based S3 method:

Detailed ingestion instructions are beyond the scope of this article. However, for those interested, more information can be found in the links included in the Library below.

What Does It Look Like?

Let’s look at an event that highlights the ‘CreateKeyPair’ action against the IAM service. A raw JSON CloudTrail event looks like the following:

{"requestParameters": {"keyName": "BOTS"}, "userAgent": "signin.amazonaws.com", "eventVersion": "1.05", "eventName": "CreateKeyPair", "userIdentity": {"accountId": "622676721278", "arn": "arn:aws:iam::622676721278:user/frothly_admin", "sessionContext": {"attributes": {"mfaAuthenticated": "false", "creationDate": "2018-07-26T20:56:40Z"}}, "accessKeyId": "ASIAZB6TMXZ7GUK5FY3H", "principalId": "AIDAIUCPPDM575G2FVOTI", "userName": "frothly_admin", "type": "IAMUser", "invokedBy": "signin.amazonaws.com"}, "responseElements": {"keyFingerprint": "02:36:de:d2:6a:97:c5:a7:f2:e7:b2:d8:ce:95:a7:bd:85:a6:af:39", "requestId": "477c8411-ae6b-4be6-9756-6cd1a29f503b", "keyName": "BOTS", "keyMaterial": "<sensitiveDataRemoved>"}, "recipientAccountId": "622676721278", "eventSource": "ec2.amazonaws.com", "eventTime": "2018-07-26T21:08:28Z", "requestID": "477c8411-ae6b-4be6-9756-6cd1a29f503b", "awsRegion": "us-west-1", "sourceIPAddress": "12.196.122.120", "eventType": "AwsApiCall", "eventID": "597b5cd4-9f1d-4087-a9e0-7a9e73d9aa3a"}

Splunk parses this raw data, extracts the CloudTrail fields, and provides a syntax-highlighted structured view of the raw JSON event:

Looking at this event, we are quickly able to see the following:

Who: The event was performed by the ‘frothly_admin’ IAM user
What: The event contains details about an attempt to create a key called ‘BOTS’
Where: The action to create a key pair occurred in the ‘us-west-1’ AWS region
When: The event occurred on ‘2018-07-26’ at ‘21:08:28Z’ (UTC)

Further details about the action can be found in additional exciting fields, such as:

• The attempt to create the key was successful, noted by the lack of an errorCode field     
• The key was created by a user logged into the console 
      - invokedBy: signin.amazonaws.com
• The user logged into the console without using Multi-Factor Authentication (MFA)
      - mfaAuthenticated: false
• The key fingerprint is ‘02:36:de:d2:6a:97:c5:a7:f2:e7:b2:d8:ce:95:a7:bd:85:a6:af:39
      - keyFingerprint: 02:36:de:d2:6a:97:c5:a7:f2:e7:b2:d8:ce:95:a7:bd:85:a6:af:39 
•  AWS does not include the private key information in the CloudTrail event
      - keyMaterial: <sensitiveDataRemoved>        

INTERESTING CLOUDTRAIL FIELDS

eventName

Of particular interest is the ‘eventName’ field. This field contains the requested action, which is one of the actions in the API for that service. After sorting this field by count, subfields using a regex, we can further identify individual actions and the corresponding services:

Here we can see that the AWS ‘Config' service requests a large number of ‘Describe Evaluation Status.' We'll learn more about Config and its data in a later blog post. For now, the various types of events within the CloudTrail data—the actions (Get, Describe, Run, etc.) and services (Config, Instances, Bucket, etc.).

Security Use Cases

A common goal of security operations in any environment is to detect signs of credential compromise. CloudTrail can be especially useful in this capacity. Since an ounce of prevention is worth a pound of cure, let’s first look at a way to identify signs of increased risk of compromise. An industry-standard document on AWS security best practices that your organization can use to establish security policies and requirements is the CIS Benchmark. Among the many recommendations, CIS recommends the use of Multi-Factor Authentication (MFA) on accounts with a console password (Section 1.2) and root accounts (1.14). Enabling MFA helps secure accounts, so conversely, the lack of MFA may result in accounts that are more easily compromised.

Console logins appear in CloudTrail with an eventName of ‘ConsoleLogin.' The use of MFA for console logins appears in the ‘additionalEventData' section. Combining these two attributes into a Splunk search, we can identify users logging into the console without MFA:

sourcetype=aws:cloudtrail eventName=ConsoleLogin
| stats count by username, additionalEventData.MFAUsed

We can see that user ‘bstoll' has logged into the console four times without using MFA. Hopefully, this represents a policy violation within your organization that would warrant follow-up with bstoll or his administrator to enable MFA on the console account. Thankfully we do not see any signs of root login without the use of MFA. This search can easily be saved as an alert that a SOC could take action on.

Let’s take a look at how CloudTrail can be used not for proactive alerting, but for security hunting. Let’s look at some of the errors associated with Service actions. A simple search shows the various error messages:

sourcetype=aws:cloudtrail
| stats count by errorCode
| sort - count

Most of the API actions were successful. While compromised account activity may result in successful command execution, let's look at the lower-hanging fruit. There are 7 events with an errorCode of ‘AccessDenied.' That seems interesting enough to look at further. Filtering on those events, and formatting the output into a table shows some interesting results:

sourcetype=aws:cloudtrail errorCode=AccessDenied
| table awsregion, eventName, userName, src_ip, userAgent, errorMessage

From this view, we can see the user ‘web_admin' is attempting many actions that are ultimately denied. The errorMessage indicates some suspicious activity - what's with all that ‘nullweb_admin' activity? Could the ‘web_admin' user possibly be compromised? Should we see activity from those source IPs? It's worth a follow-up from the SOC team to find out what's going on here.

CloudTrail and Splunk

In addition to Splunk’s AWS data ingestion capabilities, the Splunk App for AWS provides multiple out-of-the-box views into CloudTrail data for security-relevant services such as IAM user and key activity, S3 buckets, Config policies, and much more. Below is a view into key pair activity:

Library

• CloudTrail Record Contents
• CloudTrail ‘userIdentity’ Element
• Splunk Add-On for AWS
• Splunk AWS Input Types