SQL + Splunk = SplunkMSE

Introducing SplunkMSE (Splunk MySQL Storage Engine).

SQL is the lingua franca of structured data.  Likewise, Splunk is the way to work with highly unstructured data generated in the data center.  Data residing in relational databases can be analyzed via a plethora of off the shelf tools like Excel, Tableau, Cognos, Crystal Reports and on and on.   SQL is well known by developers everywhere. What better idea than using these tools to work with data that lives within Splunk?

SplunkMSE is fully open source. Visit SplunkMSE’s home site  for downloads, installation instructions, detailed documentation, source code and more. While there, I encourage you to ask questions, file bugs and if the overwhelming urge to fix them should arise, feel free to do so.

To see a brief introduction of SplunkMSE, check out this 6 minute video.

A Bit More Info

SplunkMSE allows Splunk to be used as a data storage back end for the MySQL RDBMS.  What this means is that SQL, Tables, Databases, ODBC, JDBC and buzzwords well-known in the Relational Database community can now be used to access data that lives within Splunk.  No importing or exporting required.

One thing that’s really interesting about this project is the core idea of integrating the following concepts:

• Early Structure Binding – Requiring predetermined structure at insert time.  This is how relational databases generally work – you insert data into predefined table structures.  This required structure is great in many cases, but not for IT data with 1000’s of different event types which are constantly changing.
• Late Structure Binding – Structure is applied at query time on the results of that query, but no structure is required at insert time.  This is how Splunk works – it doesn’t matter what the data look like when you index it, but structure is derived using automatic and user-configured heuristics after you filter the data with a search.

What this means to SplunkMSE is that we can’t create tables until we execute the search to figure out the structure.  So, to make a long story short – SplunkMSE allows the user to map Splunk searches to MySQL tables.  These tables are created by executing a Splunk search, analyzing the structure of the results and using that structure to create a table.  When the user queries the table with a SQL SELECT statement, the associated Splunk search is executed and the results are dropped into the columns of the table.

SplunkMSE provides a simple user interface for creating and editing the mapping of Splunk searches to MySQL tables along with a few other basic tools.

Enjoy.

Podcast: Play in new window | Download