Making Sense: Manufacturing, Splunk and Industrial Data

Recently, in the online publication Manufacturers Monthly, Denise Carson published a piece called “Harnessing Operational Intelligence”, and really made the case for using big-data and platforms like Splunk to deal with “rising costs and the tyranny of distance”. Denise explained that operational intelligence has the potential to help manufacturers do things smarter and remain competitive in the face of massive volumes, velocity, and variety of data.

In the same week, in the “Smart Business” section of the Chinese language, Yu Zhihao wrote about how a Korean semiconductor company was using Splunk and big data to perform real-time analysis of the semiconductor production line, and was quickly getting to the bottom of production issues through advanced analytics and real-time alerting.

In that article (open in Chrome to translate from Chinese), details are presented on what is really a very advanced application of Splunk. As you may know, Splunk Enterprise is a highly extensible machine data platform. Just as Splunk’s core framework was extended for applications including Enterprise Security, and PCI Compliance, customers and partners are now recognizing that by harnessing the core capabilities of Splunk, but by providing some code extension and domain expertise on top of the platform, you can build world class solutions for specific domains, even in emerging areas like the internet of things, industrial environments and manufacturing. Lets look a little bit further into how this specific manufacturer built a Manufacturing Management Solution with Splunk.

diagram Splunk usecase manufacturing management

First, the semiconductor manufacturer uses Splunk to monitor 300GB of daily log data from the Tibco middleware already installed in their environment. While it appears that this data is loaded through file system monitoring and a Universal Forwarder, loading data from TibcoEMS and other message queues could also be done in real-time with the JMS Messaging Modular Input. There are other Message Queue Modular inputs built for Splunk to index data from Amazon Kinesis, Apache Kafka, MQTT, SNMP, and AMQP, please check out our IoT Solutions webpage for more information on accessing data through these methods. You can also easily access the real-time industrial data from manufacturing equipment through the Kepware Industrial Data Forwarder for Splunk.

Splunk ingests this data into a tier of indexers, and a set of rules (close to 300 according to the article) run in Splunk to identify abnormal manufacturing process steps and alert facility operators of the issue. Operators can then re-adjust the machine parameters to improve yield (2% reported) and to reduce raw material cost by 5%.

In another facility, an integrated circuit (IC) packaging and testing plant, quality assurance (QA) data from the testing equipment is entered into Splunk from a summarization layer for search, exploration and analytics.

diagram big data qa platform

Through a brilliant extension of Splunk with the R Project for Statistical Computing (see the R Project App for Splunk) and SciKit-Learn, novel complex event processing and machine learning capabilities were built into the solution to enable advanced statistical processing of the QA data. Insights gained through this approach apparently reduced IC packaging defect rates by 5%!

As Splunk continues to be applied to novel applications, there will be a continued demand for development on Splunk’s machine data platform to provide solutions based on specific domain expertise. Manufacturing is just one of the areas where we are seeing interest and acceleration of Splunk adoption. Oil and Gas Production, Smart Buildings, and Transportation are all areas where operational intelligence is gaining momentum, and there will be clear demand for applications and solutions built for these industries. As Denise Carson noted, Splunk and operational intelligence has the potential to truly break down data siloes and provides “a pragmatic approach” for industries to “optimize and innovate for sustainability and competitive advantage”.

As a catalyst to innovation in all areas of operational intelligence, Splunk is holding the Apptitude contest, an online competition for the next big app in Splunk. I’d love to see someone build a market changing application for industrial data and win this thing. Think of the possibilities – automated fault detection and diagnosis? Some advancement in rules processing? A unique visualization platform for industrial environments? The possibilities are only limited by your imagination. And to spark that imagination, and a bit of action, we are a offering 1st prize of $20,000 cash and a free trip to .conf2015. Competition will be stiff, lets see what you have!

Any questions, concerns, thoughts, or results sets from general musings? I can always be reached at, through our IoT and Industrial “Ask the Experts”, on Twitter @BrianMGilmore, or on LinkedIn at

Brian Gilmore

Posted by