Over the past few years, I have had the opportunity to talk to so many customers using Splunk to manage their AWS environment. I have always heard the same thing- give us more! I was at AWS re:Invent 2013, when we launched the first version of the Splunk app for AWS, and I was impressed with the excitement around the dashboards and insights we offered for AWS CloudTrail. Since then we have expanded to fourteen source types (AWS CloudTrail, ELB Accesslogs, S3 Accesslogs, Cloudfront, Billing, Cloudwatch, Cloudwatch logs, VPC Flowlogs, Inspector, Config, Config Rules, Description & Config Notifications) and three inputs to collect data (S3, SQS & Kinesis). In that same time, we have also grown from a few to over forty dashboards.
AWS re:Invent 2016 is this week and we’re thrilled to announce the Splunk app for AWS 5.0. In this post, I want to cover some of the cool new features we added in this release. As customers move to the cloud, they are trying to maximize their value as they tap into the infinite compute of cloud computing. Amazon has made it so easy to tap their services that we find our customers looking to build out a governance model for their account management. Because of this explosion of growth, we have chosen to concentrate this release on helping account owners get the most from the services they use in AWS.
Reserved Instance Planner
Of all the features in this release, I am most excited about the Reserved Instance Planner. With this set of dashboards we collect your current usage via the described API or better yet, the Detailed Historical Bill from your account. With this data we are able to suggest the number of RIs for each instance type you should purchase for the following year – assuming consistent usage – and tell you how much money you will save by reserving them.
This experience is also interactive. You can manipulate the graph by pulling the lines to adjust the recommendations. This feature will enable you to consider things like seasonal load, or the future growth you are expecting in your infrastructure. I really think you will love this feature.
Reserved Instance Inventory
The Reserved Instance Inventory dashboard gives you a simple way to track your previously purchased reserved instances. It lets you see all your sub-accounts on one single dashboard, which you can’t do in the AWS console today. You can even filter by Instance Type, State, Duration, and Region. On this dashboard you can even see how much you paid for the instances and when they are expiring. This dashboard helps you manage your accounts and ensure that RIs you have purchased to control cost do not expire without your knowledge.
In our previous version of our app we created a feature we called the Recommendation Engine. It used machine learning to recommend upgrading and downgrading the EC2 instance to better utilize the hardware based on CPU utilization and network traffic. This was very popular with our users so we built upon this feature and renamed it Insights. We now offer insights for EC2, EBS, EIPs, and ELB insights. These Insights are based on best practices and experiences we had internally at Splunk. Insights will both help you save money and improve your efficiency running your own services in AWS.
Anomaly Detection for CloudTrail & Billing data
One thing we have heard from many of our customers is that they want to be enabled to configure their own “Insights”. So we integrated machine learning and created a dashboard to configure your own CloudTrail and Billing data insights. Anomaly insights are driven by machine learning toolkit. To use this go to the anomaly dashboard and choose Account, Event Name, Response, and a Granularity. Once you save it you have basically enabled machine learning monitoring on your account. From here it will start trending how your account has been used in the past and will call out Event Types that are happening at a higher rate than they usually do with normal usage. For example, this is really useful for identifying account compromises or runaway provisioning where a larger number of instances are spun up that happen in normal account usage.
Our topology dashboard has been very popular with customer looking to visually explore their accounts. We use AWS config data to create a relationship based way to visualize your AWS infrastructure.
Not only haven we greatly improved the performance on the dashboard, we have also added support for Elastic Load Balancers, and more Insights.
Timeline is a new dashboard and a concept in our journey to democratize AWS time series data. You can think of this as your “social media” timeline for your EC2 instances. In this dashboard you can choose an instance, and pick from any of the four or all of the data sources (Config, Inspector, Config Rules or CloudTrail Events). The Timeline gives you a new tool to track down compelling Events in this time series visualization.
Server-less architecture is surging in popularity and we have heard from you and acted. We are introducing new Lambda dashboards. With this dashboard you can track duration of running functions, see if you are being throttled, and even track the amount of data transferred during a function. I know you will find them useful.
Enterprise Account Governance
As I stated above, we have concentrated on larger account management, collecting and displaying larger datasets. We have also improved our documentation around best practices for isolation of account and source type specific data.
More New Out of Box Alerts
To get you started even faster we have authored alerts to enable a bunch of our new features. We have alerts to notify when you have Billing or CloudTrail Anomalies, and even one to notify you when your RI’s are going to expire in the next month.
We hope you find the features in the new app as useful as we do internally. If you want to try out the Splunk App for AWS you can download Splunk, or start a Cloud or Light Cloud trial at Splunk.com. Also, check out the Splunk app for AWS in Splunk Light starting at $3 dollars a day on the AWS Marketplace.
Principal Product Manager