Resilient by Design: The Role of AI and Security in Observability for Plant Operations

In today’s rapidly evolving manufacturing landscape, observability has become a cornerstone for resilient and efficient plant operations. Manufacturing plants, often characterized by a mix of legacy equipment and complex, interconnected systems, face unique challenges in maintaining high levels of uptime and reliability. The stakes are high, as unplanned downtime can cost manufacturers up to $225 million per year — impacting not only revenue but also customer trust and supply chain commitments, according to Splunk's The Hidden Costs of Downtime report.

Zero downtime has always been expected, but as plant operations grow more complex, advanced observability and end-to-end visibility into every layer have become indispensable. This requires more than just traditional monitoring; it demands proactive issue identification and rapid response to anomalies before they escalate into costly disruptions.

With the growing adoption of advanced digital technologies such as IoT, AI/ML, and cloud computing, plants are transforming the way they operate and respond to dynamic challenges. However, with these advancements come new vulnerabilities and security concerns. The convergence of IT and OT systems expands the attack surface due to increased interconnectivity, making security a critical component of any observability initiative.

Ready to elevate your observability strategy and drive digital resilience? Let’s explore how Cisco and Splunk solutions can transform your plant operations.

Splunk AppDynamics

Splunk AppDynamics stands out as a powerful observability platform well-suited to the unique demands of manufacturing plants. While more and more organizations are migrating or considering migration to modern architectures such as microservices or service-oriented architecture (SOA), many industrial environments still rely on a mix of home-grown and commercial off-the-shelf (COTS) applications built as monolithic systems that can be difficult to monitor and optimize. Splunk AppDynamics provides deep, real-time visibility across these complex application landscapes, enabling operations teams to quickly pinpoint performance bottlenecks and security issues before they impact production. Splunk AppDynamics uses AIOps capabilities to automatically detect any performance issues in your applications and infrastructure. It helps you to quickly identify the root cause of an issue and troubleshoot it. The following capabilities are available:

• Dynamic Baseline - Splunk AppDynamics establishes dynamic baselines for application performance, automatically adapting to changes in normal behavior patterns over time.
• Anomaly Detection - Splunk AppDynamics detects anomalies in real time, alerting teams to unusual activity or performance issues as they arise.
• Automated Root Cause Analysis - Splunk AppDynamics quickly identifies the root cause of performance problems, reducing the time required for troubleshooting.
• Automated Transaction Diagnostics - Splunk AppDynamics delivers automated diagnostics for business transactions, pinpointing issues and performance bottlenecks across complex workflows.

Splunk AppDynamics when combined with Cisco Secure Application, reduces the risk of security exposure without compromising application delivery speed. Traditional vulnerability scanning typically occurs before production deployment and at set intervals, leaving newly launched applications exposed to emerging threats and zero-day exploits. Cisco Secure Application enables continuous vulnerability assessment and protection by monitoring code execution in real time, allowing for immediate detection and prevention of possible exploits. With real-time vulnerability monitoring and business risk scoring, manufacturers can proactively prioritize and address security risks, supporting resilient, secure, and zero-downtime operations.

Practical Implementation: AIOps in Observability for MES Application

In this section, we’ll explore how AIOps in observability can be leveraged in a sample Manufacturing Execution System (MES) application. Let’s consider the home-grown MES platform responsible for dispatching machines connected to stations alongside manufacturing lines. While the machines themselves and the underlying PLC layer are outside the scope of this scenario, enhancing visibility at the control plane can still provide valuable insights. Monitoring the control logic and communications within the MES enables the Plant Operations team to proactively identify certain potential issues, optimize dispatch processes, and take preventive action to minimize the risk of downtime. However, it is important to note that MES visibility alone may not always reflect problems occurring at the machine or PLC layer, which can impact line operations despite apparent MES availability.

Our sample MES application is fully instrumented with Splunk AppDynamics, enabling comprehensive Flow Map visualization and access to essential KPIs. With this instrumentation, teams can easily monitor application workflows, track performance metrics, and quickly identify bottlenecks or anomalies in real time. The visual Flow Map provides a clear understanding of how different components within the MES interact, making it easier to pinpoint issues and optimize dispatch operations across the manufacturing floor.

Each machine is managed through a separate business transaction within the MES application, often involving multiple interconnected components. These transactions may include interactions with cache systems, databases, and other critical enterprise applications such as QMS, SAP-ERP or CRM. By tracking these business transactions end-to-end, Splunk AppDynamics provides deep visibility into the flow and performance of each operation, making it easier to detect dependencies, identify potential bottlenecks, and ensure seamless integration across systems.

While monitoring the overall health of business transactions and leveraging health rules can alert teams to potential problems, minor issues that could eventually escalate are sometimes overlooked. This is where the Anomaly Detection feature in Splunk Splunk AppDynamics becomes invaluable. By continuously analyzing performance data and identifying subtle deviations from normal behavior, Anomaly Detection can surface early warning signs that might otherwise go unnoticed. This proactive approach enables operations teams to address small irregularities before they compound into larger, more disruptive problems, further strengthening the resilience of plant operations.

Let’s look at one example of Anomaly Detection in action. At 10:19 a.m., the Plant Operations team receives an email notification alerting them to an anomaly detected in the MES application. Upon logging into the Splunk AppDynamics controller, they can immediately confirm that an anomaly event has been recorded.

As the Plant Operations team investigates the incident, Splunk AppDynamics quickly highlights a suspected cause for the anomaly - an increase in response time associated with a particular machine. By pinpointing this specific area, Splunk AppDynamics enables the team to focus their investigation where it is most needed, accelerating the process of identifying and addressing the underlying issue. However, to ensure a thorough root cause analysis, the team may need to look into data from both machine and network layers, recognizing that problems can also originate elsewhere in the supporting infrastructure. Whilst not having the full picture yet, this targeted and layered approach helps minimize potential disruption to plant operations and allows workers to perform their tasks more confidently and accurately — ultimately leading to fewer mistakes and a safer, more productive environment.

This example demonstrates the effectiveness of AIOps capabilities within Splunk AppDynamics. Rapid detection and diagnosis of anomalies help reduce downtime in manufacturing environments, while early identification and resolution of issues safeguard operational continuity and enhance overall plant resilience. For even deeper insights and a complete view across both machine and network layers, integrating with Splunk Platform and Cisco ThousandEyes can further empower teams to resolve incidents quickly and maintain optimal plant performance.

Practical Implementation: Security in Observability for MES Application

In this example, we will explore how real-time vulnerability monitoring can aid in the early detection of security incidents and help minimize operational risk. By continuously assessing applications for potential threats, manufacturing teams can respond proactively to vulnerabilities, reducing the likelihood of disruptions and enhancing the overall security posture of the plant.

Let’s consider the following scenario. During a recent rollout of a new version of the client-side MES application, a comprehensive vulnerability scan was performed, and the app was confirmed to be free of known issues at launch. However, threat actors are constantly evolving their tactics, searching for new vulnerabilities and potential entry points. Even with thorough pre-production testing, the risk remains that newly discovered or zero-day exploits could be targeted soon after deployment.

The Plant Operations team was alerted to a sudden increase in the Business Risk Security score for the Buttercup-MES application. This real-time notification signaled a potential vulnerability or security incident, prompting the team to investigate further and take swift action to mitigate any associated risks to plant operations.

Cisco Secure Application provides several in-depth views where all detected vulnerabilities are clearly listed, enhanced by integration with Cisco Vulnerability Management and threat intelligence from Cisco Talos. Each vulnerability can be thoroughly investigated to assess its potential impact on operations and review recommended remediation steps. In many cases, remediation may involve upgrading to a newer, more secure version of the affected component. These detailed insights, powered by Cisco’s robust vulnerability management and threat intelligence capabilities, empower teams to efficiently prioritize and address vulnerabilities, reducing security risks and supporting continuous plant resilience.

These comprehensive details enable the Plant Operations team to coordinate with development on remediation priorities and actions. Notably, Cisco Secure Application detected a new vulnerability a few days after production deployment. Pre-production scanning could not have caught this issue because the vulnerability was only disclosed after the deployment. By sharing vulnerability specifics and recommended remediation, such as upgrading to a safer version, both teams can quickly address emerging risks and ensure continued security for the MES application.

To make it even more effective, Cisco Secure Application can be integrated with Splunk Enterprise Security (Splunk ES). This powerful combination enables security teams to centralize the detection, analysis, and response to runtime application security events (such as vulnerabilities, exploits, and suspicious activities) directly within the Splunk ES platform. By leveraging Cisco’s advanced threat intelligence alongside Splunk’s comprehensive analytics and correlation capabilities, organizations benefit from a unified, real-time view of their security landscape. This integration streamlines incident response, enhances threat prioritization, and helps teams remediate risks more efficiently across critical application environments.

The Art of Possible

Splunk AppDynamics can serve as a key component within a broader observability architecture for manufacturing environments. When combined with the Splunk Platform, organizations gain powerful capabilities to collect, analyze, and visualize data from across IT and OT domains. By integrating with other Cisco solutions such as Cisco Cyber Vision for deep OT visibility, as well as additional available apps, manufacturers can unify monitoring and security across their entire operations. This holistic approach empowers teams to detect and resolve issues faster, enhance their security posture, and drive continuous improvement. By bringing together insights from applications, infrastructure, and industrial systems, manufacturers can unlock new levels of resilience, efficiency, and innovation in today’s connected manufacturing landscape.

Summary and Call to Action

For manufacturers today, achieving resilience starts with comprehensive observability, strong security measures, and data-driven analytics. Solutions like Splunk AppDynamics, combined with the Splunk Platform and Cisco’s broader ecosystem enable manufacturers to gain unified, real-time visibility across both IT and OT environments. By leveraging AI, machine learning, and proactive security monitoring, organizations can reduce downtime, mitigate risks, and drive operational excellence.

Now is the time for manufacturers to assess their observability strategy and embrace the next generation of integrated solutions. By doing so, you can strengthen plant resilience, secure critical systems, and unlock new opportunities for innovation in a rapidly evolving industrial landscape.

Ready to take your plant operations to the next level? Check out the Splunk for Manufacturing section on the Splunk website to discover how Cisco and Splunk solutions can help you build resilient, secure, and future-ready manufacturing.