By Splunk July 13, 2013
Splunk recently announced the beta release of Hunk: Splunk Analytics for Hadoop. As a security practitioner, this new product has some exciting implications.
For some time, security practitioners have desired to store large volumes of data, in case it would ever be needed for incident response, (anti-) fraud investigations or other uses. In an ideal world, you’d have six months to a year’s worth of data stored for investigations, however the realities of SAN costs only make it realistic to have maybe 30 days worth of data stored.
With the arrival of Hadoop several years ago, there was finally a cost effective option for storing large volumes of data on commodity hardware. The only issue is that Hadoop is primarily a storage solution, not an analytics solution. While Hadoop components can perform analytics operations in batch-mode, those components are difficult to use
The beta release of Hunk spans this chasm. Hunk creates and manages virtual indexes for Hadoop providing interactive data analysis. In other words, security practitioners can now have their “cake” and “eat” it too. Organizations can finally store large volumes of sensor log data in a cost effective manner and still be able to analyze that data easily.
Being able to combine security analytics with operational analytics is increasingly important for security reasons. The reality of information security today is that security-relevant data is found not only in security product logs (e.g., firewalls, IDS / IPS, anti-malware, etc.) but also in operational IT system logs (e.g., routers, load balancers, applications, etc.).
Looking forward, being able to combine these data sources to identify security-relevant data will only become more important as the Internet of Things increasingly pervades our lives and organizations. The need to store and analyze ever increasing amounts of data in near real time is quickly moving from a best practice to a requirement. Hunk is helping organizations keep pace with that reality.