Leveraging Machine Learning, Splunk Enterprise, the Splunk Edge Hub and MQTT for Data Integration

Enhance your operations and decision-making processes through technology and automation. Machine learning and AI have emerged as transformative forces, capable of extracting valuable insights from raw data. With the combined power of the Splunk Platform and Cisco Meraki one can leverage existing hardware to build new datasets and insights. Utilizing a Meraki MV smart camera with RTSP feeds enabled companies to unlock new potential in their data collection and analysis efforts. Utilizing cameras, originally designed for security and surveillance, can also be used to capture a diverse array of data, facilitating innovative applications across various industries.

The Meraki MV cameras provide a unique advantage to allow for easy integration into machine learning workflows as they optionally provide RTSP feed capabilities. This feature allows businesses to integrate machine learning algorithms to analyze video and audio feeds, transforming them into actionable insights. From monitoring customer behavior in retail settings to enhancing security protocols with intelligent detection systems, the possibilities are vast. By utilizing machine learning with existing camera hardware, businesses can maximize their technological investments, reducing the need for additional infrastructure while expanding their data-driven capabilities.

Figure 1-1 – Through the use of cameras and machine learning, the Splunk Platform helps you get a bird's eye view of your telemetry.

Machine learning applied to existing technologies including the Meraki MV smart cameras can also offer significant value in niche applications. For example, the use case of identifying bird species can demonstrate how machine learning can be easily applied to environmental monitoring. This example use-case highlights the adaptability of machine learning solutions, showcasing their potential to provide valuable insights in unconventional scenarios. Through processing RTSP feeds with machine learning models and the Splunk Platform businesses can gain insights into patterns and events that were previously difficult to quantify, thus opening new opportunities for data utilization.

Have you ever wondered which three little birds are singing sweet little tunes by your office window? One approach involves utilizing the Splunk Edge Hub and its MQTT broker to get data into Splunk Enterprise. In this article we will explore using the Splunk Edge Hub in conjunction with machine learning tools to identify birds and pass this information along to the Splunk Platform. This integration enables a robust framework for capturing, processing, and analyzing data from diverse sources, enhancing the decision-making capabilities of organizations. In today's data-driven world, the ability to seamlessly integrate various data sources into a centralized system is crucial for businesses aiming to gain actionable insights or just answer the question of how many times did the crow call today?

To learn which bird were singing, I utilized a project called BirdCAGE, an open-source project available on GitHub . This project uses machine learning designed for bird call analysis. The project captures audio data from RTSP video streams and processes it using machine learning algorithms. If a bird is identified it sends a message to the Splunk Platform, where we can transform it into useful insights. This is enabled through the publication of an MQTT message to an MQTT broker, in my case the Splunk Edge Hub as it easily integrates with the Splunk Platform.

We can deploy the BirdCAGE containers onto the Splunk Edge Hub to collect audio, video and sensor data from the environment. In my use case, I hosted the containers on my laptop and utilized cameras with an RTSP feed.  Many cameras including the Meraki MV camera models have optional RTSP feeds. As data is captured, machine learning processing capabilities are applied to convert raw data into structured formats. The processed data is then published using MQTT a lightweight, publish-subscribe network protocol. The Splunk Edge Hub, with its MQTT Broker enabled, acts as the intermediary, facilitating the transmission of this data to the Splunk Platform where insights can be transformed into actions.

Value Added:

• Real-time Processing: The lightweight MQTT protocol reduces overhead, ensuring quick and efficient data transmission, vital for scenarios where real-time data processing is critical.
• Comprehensive Analytics: With data in the Splunk Platform, organizations leverage powerful analytics and visualization tools like Splunk ITSI to gain insights, uncovering trends, patterns, and anomalies.
• Enhanced Decision-making: Integrating Machine Learning data enriches analytics, offering a broader perspective on environmental factors and operational metrics, empowering informed decision-making and process optimization.
• Efficient Data Transport: Hosting machine learning detection containers and the MQTT broker on the Splunk Edge Hub ensures efficient and reliable data transport, minimizing latency and maximizing throughput, crucial for real-time processing.
• Scalable Data Collection: The integration between Splunk Platform and Splunk Edge Hub, allows for scalable data collection, accommodating growing data volumes without compromising performance.

The utilization of the Splunk Edge Hub to host the machine learning detection containers and the MQTT broker offers several advantages. It ensures efficient and reliable data transport, minimizing latency, and maximizing throughput. This is particularly beneficial for scenarios where real-time data processing is critical. The seamless integration between the Splunk Platform and the Splunk Edge Hub allows for scalable data collection, accommodating growing volumes of data without compromising performance.

With the data in the Splunk Platform, organizations can leverage Splunk's powerful analytics and visualization tools such as ITSI to gain insights. The Splunk Platform provides a comprehensive suite of features for searching, monitoring and analyzing machine-generated data. The integration of machine learning data enriches the analytics capabilities of Splunk Enterprise, offering a broader perspective on environmental factors and operational metrics. This holistic view empowers businesses to make informed decisions, optimize processes, and drive innovation.

Take Intelligent Action on Insights

With data integrated into the Splunk Platform and analyzed using powerful machine learning algorithms, one of the next steps may be to translate insights into actionable outcomes. In my use cases I built dashboards to track which birds were calling and at what time of day they were doing so. This let me know when the American Robin was singing in the morning sun. It also led to observing that as the environmental conditions changed so did the frequency of bird calls, when it was raining, very hot or after the sun set.

Figure 1-2 A Splunk Enterprise Dashboard with panels containing information about which birds were calling from a location at a specific time.

While which birds are outside your window may not bring business value, the same process can be applied to identify use-cases that are important to you and your organization. The Splunk Platform allows users to build what they need, combining data from almost any source that can be ingested. If you have a specific business need to solve and this sounds interesting visit Edge Hub Central to learn more about the Splunk Edge Hub. If you already utilizing eclipse as MQTT broker, you can monitor and log your messages by following this tutorial on Splunk Lantern.

Key Feature:

• Custom Use Cases: Build solutions tailored to your business needs by combining diverse data sources.
• Splunk Edge Hub: A powerful integration tool that supports modern data environments for seamless monitoring and analysis.
• Splunk Platform: Integrate with already existing data sources such as the Eclipse MQTT broker, logs and industrial sensors letting you build a solution for your organization
• Better Together: The combination of Meraki Cameras, Cisco Hardware and the Splunk Platform let your imagination be one of the only limits on these platforms can provide.

Why Splunk?

Get all your ducks in a row by combining machine learning, the Splunk Platform and the Splunk Edge Hub, representing a sophisticated approach to data integration and analysis. This synergy enhances the ability to monitor and analyze diverse data sets, supporting real-time decision-making and operational efficiency. As organizations continue to leverage data for competitive advantage, solutions like the Splunk Edge Hub and the Splunk Platform offer the versatility and power needed to observe and navigate modern data environments. Once you find an interesting use case explore it by signing up to start a free trial of Splunk Observabilty or Splunk Platform today!

This blog post was authored by Adam Schalock, a Software Engineering Technical Leader at Splunk, a Cisco Company.