Splunking AWS ECS Part 1: Setting Up AWS And Splunk

It’s no secret that Amazon Web Services is a powerhouse Cloud provider, and one of the market pioneers in Cloud operations. They do, after all, power some of the world’s biggest and most modern systems we all use and love today. It’s natural then that they attract a lot of users both big and small to deliver high quality and effective solutions. With growing user demand comes the need for new methods of visibility and intelligence. Splunk’s Data-To-Everything platform and framework are a great fit for customers looking for a flexible and scalable solution.

In this series of guides, we will build an example of how to leverage AWS’ firelens technology to route application logs running in both Elastic Container Services and Fargate stacks to Splunk for fast and easy application intelligence. With container logs in Splunk's indexing engine, we will explore some examples using Splunk Enterprise and Splunk Cloud as the user interface. In part 1, we will set up AWS with all of the required materials, roles and troubleshooting tools necessary and set up a Splunk HTTP Event Collector (HEC) to receive logs from either ECS or Fargate containers. Parts 2 and 3 will be split into two separate segments focused on either ECS or Fargate profiles, which you can choose to follow based on your platform of choice.

Without further ado, let’s get started on your AWS container journey with Splunk!

Understanding ECS And Fargate

Before we get started, it’s important to know what ECS and Fargate are, and how they differ from other orchestration platforms. First, Amazon’s Elastic Container Service is their own managed platform to make it easy to configure, deploy and run containers. ECS is generally regarded as partially managed, largely because much of the underlying configuration is performed by Amazon’s platform. The user is responsible for managing their cluster compute nodes, and a description of how containers are supposed to be run. Now this might sound a lot like Kubernetes, however the implementation is much simpler for users, and does not require the user to have any in-depth knowledge of infrastructure-as-code.

Containers and their configurations are organized in what are referred to as task definitions, along with a full user interface to help manage task definitions. Users then decide which ECS node or collection of nodes will handle the workload. This makes it easy for users who are focused on app deployment to get up and running quickly and reliably. This is the closest implementation to Kubernetes or OpenShift, but without having the need to manage the orchestration stack.


Amazon ECS & Fargate Topology (source:

Fargate on the other hand does not rely on the user defining their own compute nodes to run tasks (referred to as container instances). With Fargate, Amazon assumes the responsibility of managing the underlying infrastructure. All the user has to do is decide how their containers are configured and run. This eliminates the need for the user to worry about whether they have enough resources, and turns the process of running containers into a serverless process. 

The benefits of ECS and Fargate are significant; with less infrastructure to manage, users can deploy and scale enterprise applications faster than ever. However with convenience comes new challenges in leveraging data exhaust from applications. With the added layer of abstraction when using containers, visibility is more important and difficult. To help with this, Amazon introduced their Firelens technology which is a simple log and event router for containers. Firelens adds introspection capabilities to containerized applications and provides a mechanism for exporting meaningful data to other platforms.

Platforms like Splunk!

Getting AWS Ready

In order for the integration between ECS or Fargate and Splunk to work, we need to get a few items set up. We’ll need a couple of roles created to access AWS resources and a place to send some logging information for troubleshooting purposes. This will help us understand what’s going on if things aren’t going as planned.

First, it’s important to make note of which zone you’re operating in. For example, this guide uses the US-East-1 zone but this will vary by account. Make sure that all resources in AWS are created in the same zone! 

Setting Up A CloudWatch Log Group

First, navigate to the CloudWatch console. Here we’ll create a new log group to receive any events from our AWS configurations as a fallback if they’re not working properly.

  1. Create a new log group with a meaningful name. We’ll call ours SplunkECS.
  2. Make sure to keep the retention period short since we won’t be needing these logs long-term.
  3. Make note of the Log Group Name as we’ll be needing it when we configure our log routers in the next blog series.

Setting Up The AWS IAM Roles

We’re going to rely on two IAM roles defined in AWS to give us access to all of the resources we need throughout this series. This will be necessary to access the CloudWatch log group, storage locations where we’ll be saving our configurations, and our task executions.

Setting Up A Role To Run ECS Container Instances

Note: This role is required only if you want to run ECS containers that send data to Splunk. If you only want to use Fargate containers, this role is not required.

  1. AWS IAM roles can be managed in the IAM Management Console.
  2. In the Roles menu, create a new AWS Service role with the EC2 Use Case template. Attach the following two policies to the role:
  3. AmazonS3ReadOnlyAccess: Alternatively, you can use the “AmazonS3FullAccess” role if you intend to work dynamically with S3 buckets within your application containers. Because our applications in this series don’t need to write resources to buckets, we’ll be using a policy with ‘read only’ access for security.
  4. AmazonEC2ContainerServiceforEC2Role
  5. Save the new role as “ecsInstanceRole”. This will be used for configuration of EC2 workloads.

Setting Up A Role To Run ECS And Fargate Tasks

The second role we’ll be setting up will be used to run the ECS and Fargate tasks. This Amazon resource will be executing the workloads on our behalf and accessing the remaining AWS resources. This role will be referenced throughout the remainder of this series and is critical to everything running. 

  1. In the Roles menu, create a new AWS service role with the Elastic Container Service Task Use Case template
  2. Attach the following two policies to the role:
  3. AmazonS3ReadOnlyAccess. Alternatively, you can use the “AmazonS3FullAccess” role if you intend to work dynamically with S3 buckets within your application containers. Because our applications in this series don’t need to write resources to buckets, we’ll be using a policy with ‘read only’ access for security.
  4. AmazonECSTaskExecutionRolePolicy
  5. Save the new role as “ecsTaskExecutionRole”. This will be used for configuration of EC2 and Fargate tasks.

Great! Now that we’ve got our CloudWatch log group and IAM roles all ready to go, our initial AWS configuration is complete. We will be focusing on specifics for ECS and Fargate respectively in the following sections.

Configuring Splunk Inputs

Now, on to the Splunk configuration. Since AWS will be using Firelens to route our application logs from our containers out to Splunk, we’ll need to set up a listener within Splunk as an open channel of communication. We’ll be using Splunk’s HTTP Event Collector (HEC for short) to do this.

Detailed documentation on creating HTTP event collector listeners is available in Splunk’s user manual.

The first architectural decision you’ll need to make is where to store your data within Splunk. This can be any index of your choosing, but for the sake of simplicity, we’ll be using a new index called ‘scratch’ for this series. Our scratch index is configured as an events index because we’ll be sending unstructured log data to Splunk in the form of application logs.

If you’re unfamiliar with the process of creating indexes, please refer to the documentation which best describes your environment

Once you’ve identified the index you wish to use, make note of it as we’ll need to refer to when we create our endpoint. In the Settings menu, select the Data Inputs field.

From here, we will Add a new HTTP Event Collector object

First, we will ensure that all tokens are enabled, and make note of our protocol and port which our Splunk Search Head is configured to. The Global Settings button at the top of the main dialog will provide this information.

Create a new token and provide a meaningful name and description. Leave the rest of the settings as their default values.

Set the sourcetype, application context, allowed indexes and default index. Since we’ll be defining the sourcetype when we configure our AWS Firelens routing, we can leave this as automatic. The format of the inbound logs will depend on whatever you are running as a containerized application in ECS or Fargate.

Finalize the configuration and navigate back to the HTTP Event Collector Data Inputs page. Now we should see our newly created token! Make note of this token, as it will be required in later parts of this series.

The last step is to test the HTTP Event Collector token and ensure that we have all our values necessary to configure our AWS Firelens log router later on.

The simplest way to do this is to run a curl command from a command line.

Note: if you are working on a Windows workstation, this is a really good tutorial on working with curl.exe which includes notes on how to install and use the tool.

An example curl command using the token retrieved when we created our token (note, your token will be different):

Splunk Cloud:

curl -k -H "Authorization: Splunk fc1db82b-5310-4d44-9fb7-113be8ff045d" -d '{"event": "hello world"}'

{"text": "Success", "code": 0}

Splunk Cloud free trial:

curl -k -H "Authorization: Splunk fc1db82b-5310-4d44-9fb7-113be8ff045d" -d '{"event": "hello world"}'

{"text": "Success", "code": 0}

Splunk Enterprise:

curl -k -H "Authorization: Splunk fc1db82b-5310-4d44-9fb7-113be8ff045d" -d '{"event": "hello world"}'

{"text": "Success", "code": 0}

Since our token is tied to a default index (which we configured as ‘scratch’ in step 6), we should now see the event successfully sent to our Splunk indexer. 

index=scratch “hello world”

Huzzah! It works!

So now we should have everything ready to go to start brokering data between ECS / Fargate and Splunk.

As a matter of best practice, keep track of the information we’ll be needing in the next parts of this series with the asset and values as follows:

  • AWS Region: US-East-1 (the region you’re working in)
  • AWS CloudWatch Log Group: SplunkECS (the name of the log group) 
  • AWS ECS Instance Role: ecsInstanceRole (the name of the role to run container instances)
  • AWS Task Execution Role: ecsTaskExecutionRole (the name of the role to run ECS tasks)
  • Splunk HEC server (Splunk Cloud):
  • Splunk HEC server (Splunk Enterprise): (Splunk Enterprise)
  • Splunk HEC Port: 8088 or 443
  • Splunk HEC Index: scratch (the name of the index you configured in your HEC)
  • Splunk HEC Token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

In part 2, Splunking AWS ECS Part 2: Sending ECS Logs To Splunk, we will create an ECS cluster and deploy our first task definition which includes a simple web server and sends its logs to Splunk.

In part 3, we will create a Fargate profile and send a task definition which includes the same simple web server, but on a serverless platform and sends its logs to Splunk.

Don’t forget to check out for the latest updates, downloads and events for everything Splunk.

Happy Splunking!

My name is Andrij and I'm a data do-er. I love technology, animals, and I'm passionate about digital rights.


Splunking AWS ECS Part 1: Setting Up AWS And Splunk

Show All Tags
Show Less Tags

Join the Discussion