Getting value from your data just got really fast. Introducing Splunk Enterprise & Hunk 6.2

Update 9/27/16: As of Sept. 27, 2016, Hunk functionality has been incorporated into the Splunk Analytics for Hadoop Add-On and Splunk Enterprise versions 6.5 and later.

SW_SpeedOne of the reported challenges of data management and especially big data is the ease of getting value from the data quickly (less than 12 parsecs).

Traditionally, a lot of time has been spent collecting and preparing data. Eventually you get to ask the questions of the data, start to create the right analytics and get the insight you need from it. This can take a lot of time.



My colleague, Nima, came up with a great diagram to show this:

Where we really need to get to, is the ability to get to the “asking” stage and to get the value from the data as quickly as possible:



With that in mind, I’m delighted to announce Splunk Enterprise and Hunk 6.2 today. The product will be available for download on Oct 28th. Here’s a run through of what’s new.


Splunk Enterprise 6.2

With Splunk Enterprise 6.2 we’ve focused on three main areas of functionality:

1) Easier data on-boarding, data preparation and Advanced Field Extractor (AFX)

Thinking about the challenges of collecting and preparing data described above, getting your data, regardless of data source, into Splunk is now easier than it has ever been. There’s a very simple, wizard based interface that allows you to select if you’re uploading, monitoring or forwarding and it will guide you through the right “work flow” for getting data into Splunk. We’ve also made the data preview capability of Splunk much easier. That leads nicely on to…

AFX replaces the existing field extraction utility in Splunk Enterprise. AFX enables you to better prepare machine data for analysis. You can capture multiple fields in a single data extraction and allows you to specify the required text to filter the events you really want. You can also view the diversity and rarity of events.FieldExtraction


2) More Powerful Analytics for Everyone – Instant Pivot, Event Pattern Detection and Prebuilt Panels

Analytics for everyone

Now on to how to start analyzing the data and getting the right insights. We added Pivot and Data Models to Splunk Enterprise (and Hunk) with version 6.0. With 6.2 you don’t need to create a data model before you can use Pivot – you can create a Pivot Table (and subsequently data visualizations) straight from a search – literally with a single click. Splunk effectively creates the data model for you when you create the Pivot Table.

This works together well with another new feature – Event Pattern Detection. By automatically discovering patterns and anomalies in your data with a single click, Event Pattern Detection reduces massive data sets to their essence. This means you can see common patterns but also those events that are rare. It also means you can get instant insight from the data. In the diagram below you can see the pattern of events around food health violation and the patterns of events (10.17% of the patterns of events are moderate risk due to food holing temperature, 8.93% are low risk due to poor labeling or menu representation, etc.)


As with most things in Splunk, these event patterns are just a search (automatically generated) so you can easily turn a particular event pattern search into a Pivot Table (using instant Pivot). We’ve also added Ultra Drilldown, which means you can investigate data faster and graphically add and update underlying searches.

Finally, we’ve enhanced the ability to build dashboards much faster by assembling and reusing pre-built dashboard panels. Effectively there is now a built in repository of dashboard panels or “widgets” that you can use together with your own data visualizations. When building your own dashboard you can browse or search for all available reports, dashboards, panels etc. and preview them before adding them to your own dashboard. You can then convert them to inline so can then “clone and customise” the panel for your own purposes. You can then package these visualizations within an application to allow users to build a single pane of glass.


3) Simplified Management at Scale – Search Head Clustering and Distributed Management Console

Splunk Enterprise 6.2 adds a key capability with Search Head Clustering. This means a significant improvement in scalability, redundancy and availability of the search head. It also means that there is no single point of failure and no need for shared storage. By replicating user configuration settings, dashboards and reports across clustered search heads, Splunk Enterprise 6.2 significantly improves the number of concurrent users and searches with a uniform user experience.

To accompany this, Splunk Enterprise 6.2 has an enhanced Distributed Management Console that allows Splunk admins to easily monitor the health & performance of distributed deployments. It includes a number of new dashboards including:

  • Search Usage and Performance at Deployment-wide and Individual levels
  • Indexing Usage and Performance at Deployment-wide and Individual levels
  • Platform Resource Utilization (CPU/Memory/Disk) at Deployment-wide and Individual levels
  • “Platform Alerts” that allow the Splunk Admin to enable email alerts for pre-packaged conditions that may be detrimental to the operation of Splunk

This ships with Splunk Enterprise 6.2 with nothing extra to install.


Hunk 6.2

The good news is that Hunk 6.2 gets a lot of the new features described above from Splunk Enterprise 6.2. The Data Explorer, Instant Pivot, Event Pattern Detection and Prebuilt Panels will all be very useful when working with data in Hadoop or NoSQL data stores. There are a couple of “extras” that Hunk 6.2 gets that will bring a lot of value to managing big data.

Hunk Apps for NoSQL.

With Hunk we launched the Virtual Index technology. Whilst Splunk Enterprise indexes that data, Hunk uses Virtual Indexing to allow you to use Splunk’s search language, Pivot table, analytical tools and application development frameworks on data that doesn’t get indexed by Hunk (basically it stays where it is). The Hunk Apps for NoSQL extends Hunk Hadoop Virtual Index to allow you to access NoSQL data. These are free apps – the first two are for Sqrrl and MongoDB and you can download them and add them to Hunk for free. They make searching and analyzing data in NoSQL very straightforward and allow you to make use of all of Splunk Enterprise’s new features on top of NoSQL databases.


Hunk Sandbox & Hourly Pricing on AWS

It isn’t easy to set up, configure and run a Hadoop infrastructure just to find out if it makes sense for your organisation. The Hunk Sandbox gives you a very quick way to “try out” Hadoop and Hunk together. Not only do you take away the challenges of setting up and running Hadoop but you also get Hunk pre-installed with Hadoop to allow you to start to get the value out of your data much quicker than with Hadoop alone. If you want to try it out you can find it here.

Lastly, Hunk 6.2 is available as an Amazon Machine Image so you can leverage preconfigured Hunk instances provisioned by AWS. It is priced hourly and is preconfigured to work with data that is accessed via Amazon Elastic MapReduce. Again, this makes it really easy to start using Hadoop and Hunk together to unlock the business value of the data that is in Hadoop.


We’ve tried to make sure with these releases that the time to get the value from data and the ability to create compelling business analytics for everyone is as fast as possible. The goal is to ensure organisations get to the value from their data as quickly as possibly.

Prep2Keep an eye out for Splunk Enterprise and Hunk 6.2 on 28th October here.

In the words of a famous intergalactic IT and data professional smuggler – “Chewie. Punch It”

Posted by