By Splunk July 31, 2013
All of my posts seem to be sparked by some sort of customer interaction. The last few have been about how to do something, but this one is about what we are doing. A customer recently asked: “What can you do with Splunk in AWS?”. While there are some docs and posts that cover different tasks, there isn’t a single place to get a wide view of things. So, here goes….
First, Splunk software will happily run in EC2. As it is software that is agnostic to the operating system and hardware, EC2 is a fast way to get started with Splunk. To provide some comfort on the topic, we test and run our own Splunk instances on it. Customers with very large deployments have run the majority of their instances on EC2. For more details on recommendations, sizing, and performance, you can check the following links:
- EC2 recommendations: http://blogs.splunk.com/2012/03/07/splunk-and-aws-sizing-revisited/
- Old blog post on EC2 recommendations: http://blogs.splunk.com/2011/02/24/splunk-and-ec2/
- AWS Technical Brief: http://www.splunk.com/web_assets/pdfs/secure/Splunk_and_Amazon_Web_Services_Tech_Brief.pdf
- EC2 storage performance post: http://blogs.splunk.com/2013/06/06/splunkit-v2-0-2-results-ec2-storage-comparisons/
So you can deploy very easily and it works great, but what else? Splunk has content! Maybe you own a huge farm of EC2 instances that are used by various departments and individuals. While AWS provides accounting and usage metrics, they are not very granular. There is an App that will get you granular detail to figure out who is using and spending what, where, and when:
http://splunk-base.splunk.com/apps/65926/splunk-app-for-aws-usage-tracking
Okay, so I can now track my friends in IT to see if they really need a dozen cluster compute instances. What about the data on those systems? To go beyond our standard data gathering methods (file/directory, network, scripts), we also have a technology add-on that allows you to easily get data from S3 (Amazon’s Simple Storage Service). This means that if you are using certain Amazon services, you will have an easy way to get the data you are storing or gathering:
http://splunk-base.splunk.com/apps/64931/splunk-for-amazon-s3-add-on
So there you have it – a quick summary of what is going on with Splunk and AWS. Final note, some people have asked: “Well what about an AMI (Amazon Machine Image)?”. While there is not one yet, I’d be happy to hear comments on how useful it might be for you.
Happy Splunking!
----------------------------------------------------
Thanks!
Simeon Yep