By Splunk October 04, 2013
Introduction
In my previous blog article, I describe the practicality of collecting manufacturing data for Statistical Process Control (SPC) and Supervisory Control and Data Acquisition(SCADA) applications using Splunk. Manufacturing industries typically use OPC servers as a uniform platform to control and collect data in their production.
In this article, I will show how to integrate Splunk with OPC servers to collect real-time data, and how to integrate with databases for archived data. This is an open source project called “splunk-demo-opcda” which you can download it from splunk@github.
Architecture
The splunk-demo-opcda architecture has the following major parts:
- Modular inputs to control and manage configurations, including the RPC server, Measures and Databases.
- Customized search commands, such as dbeval, for integrating with the Splunk search engine to retrieve data from database.
- Built-in Remote Procedure Call(RPC) server to provide a platform for java programming language services.
OPC server is a DCOM/windows based application, we use Java-based, open source utgard from openscada.org to communicate with DCOM across the network.
Modular Inputs for configurations
After you download and install splunk-demo-opcda on your Splunk system, go to splunk>Manager>>Data inputs and create an RPC configuration, as shown.
Note: Currently only one instance of RPC can be running in a Splunk instance.
After creating the RPC server configuration, start it by clicking Status column in splunk>Manager>>Data inputs>>Splunk RPC Startup as shown.
OPC Server
splunk-demo-opcda use the MatrikonOPC free OPC server simulator for testing OPC connectivity. Here is the configuration needed to connect to the simulator as shown:
Services Development
Services development involves the following steps:
- Develop a client Python function to integrate with search commands.
- Develop a Java function as the client Python function counterpart to handle backend service work.
- Configure an Apache avro schema on top of the Python function as a decorator. This schema is used by the RPC server to dynamically resolve both client and server function call signatures.
Configuration example:
from service import avro_schema
import utils
logger = utils.setup_logging(“opc”)
@avro_schema(
{“runMeasure#com.splunk.opc.Test” :
dict(request=[dict(name=”server”, type= dict(type=”map”, values=”string”)),
dict(name=”items”, type= dict(type=”map”, values=”string”))],
response=”int”)
})
def runMeasure(server, items):
return runMeasure.call(server, items)
Putting It All Together
Make sure the OPC simulator is running.
Enable the measurement at splunk>Manager>>Data inputs>>OPC DA Collector.
Use splunk search command to monitor the accumulating data file: “$SPLUNK_HOME$/etc/apps/splunk-demo-opcda/json/opc.json”, for realtime data as shown in the following example:
For archived data, the customized command dbeval runs a Python function that communicates with the JDBC service in Java backend. The following example shows how the Splunk search command retrieves database entries.
The database alias “mydb” in the above search example was set up in the Database Configuration as shown:
Summary
Open source, prototype “splunk-demo-opcda” project demonstrates data acquisition from opc servers and databases, for real-time, archived data collections. The project seamlessly integrates Splunk and OPC servers, databases to provide powerful analytics functionality for easy manufacturing data collection, customization and visualization.
Acknowledgments
I would especially like to thank my colleagues, Roussi Roussev, for constant discussion, valuable suggestion throughout the development of splunk-demo-opcda.
----------------------------------------------------
Thanks!
Bill Tsay