KubeCon News: New Injector Donation, Advancements in LLM Monitoring, OpenTelemetry and AI

Since the beginning of our involvement, Splunk’s mission has been to make OpenTelemetry accessible and easy to adopt for every organization and every environment. Splunk has been a significant contributor to OpenTelemetry since its inception. Splunk has dozens of employees who work full-time on the open source project. In October 2021, Splunk made its first project donation, a metric agent for monitoring infrastructure and application services. Since then, Splunk has made other notable donations, including the eBPF Collector, SQLcommenter (in partnership with Google), and OpenTelemetry Instrumentation for Android. Today, we’re excited to share our latest donation to the project, the OpenTelemetry Injector.   

For many, this will solve the final hurdle in realizing the benefits of OpenTelemetry adoption. We’ve heard from multiple organizations about the challenges in leveraging OpenTelemetry in their observability practice. Many organizations have yet to leverage containerized platforms like Kubernetes. For these organizations, manual work was required for each application they ran in order to instrument them.  

With the OpenTelemetry Injector, those in a non-containerized environment can now enjoy a much easier setup process for OpenTelemetry. This means zero configuration for organizations leveraging non-containerized environments. It’s as simple as installing the OpenTelemetry Collector on your machine, restarting the applications you want to monitor, and voila, basic instrumentation is complete. Easy instrumentation is important, because OpenTelemetry usage has significant business benefits, beyond the obvious one of requiring less engineering effort to start an observability practice.  

OpenTelemetry: A Business Catalyst  

One of the foundational benefits of OpenTelemetry is its breadth of support for languages, frameworks, and operating systems, helping organizations build a robust observability practice no matter what their infrastructure looks like. Mature, leading observability practices go beyond simple monitoring of applications and instead act as a core business function that provides access and business context to all of a company’s data. This understanding leads to business impact, as seen in recent Splunk data connecting observability directly to business outcomes such as employee efficiency and customer experience.    

According to data from Splunk’s 2025 State of Observability report, which surveyed more than 1,855 ITOps and engineering professionals:  

• 74% of observability practitioners indicate observability has a positive impact on employee productivity 
• 69% of observability practitioners indicate observability has a positive impact on customer experience 
• 65% of observability practitioners indicate observability has a positive impact on overall revenue  

The data also indicates that use of OpenTelemetry within an observability practice can act as a business catalyst. Organizations that have adopted OpenTelemetry indicate they are seeing positive effects on their revenue growth (72%), operating margins (71%), and brand perception (71%) specifically as a result of their OpenTelemetry adoption.  

This data not only aligns with the vision we had when we began supporting OpenTelemetry at its inception but was a key reason we donated the OpenTelemetry Injector. We want organizations to understand their applications, their data, and their systems so they can develop a complete understanding of how to maximize their business. Further, we believe data ownership is a fundamental right – any organization should be able to control what data is collected, how it is processed, and where it is analyzed and stored. OpenTelemetry enables this flexibility in a vendor-neutral yet powerful way.  

OpenTelemetry and LLMs    

In addition to business growth and impact, OpenTelemetry also has future-proofing capabilities to help organizations maximize ROI for AI.   

While LLMs help organizations remove manual tasks and streamline everyday business processes, they currently lack easy visibility, leaving everyone — from extremely technical experts to business leaders — in the dark about how LLMs are coming to their conclusions. Currently, there is a lot of work in the observability community to understand LLM runtimes and the inner workings of generative AI. OpenTelemetry has been on top of this, creating semantic conventions and other standards for instrumenting AI applications.  

In September, Splunk and AGNTCY, a collective dedicated to the discovery, identity, messaging, and observability of AI agents, partnered to donate a schema to OpenTelemetry that allows users to emit data about LLMs in a consistent way, making it easier to analyze and get insights from. Additionally, the collective has developed a Metrics Compute Engine for AI applications, providing a way to calculate key LLM performance and reliability metrics and to emit those metrics in an OpenTelemetry-compliant format to observability platforms. OpenTelemetry-powered applications are quickly able to analyze LLM outputs on multiple levels, including qualitative metrics (such as right answers and bias) and quantitative metrics (such as cost and time to respond).  

Splunk Observability Cloud can use this data to show key application data like latency, errors, and request rate, and also AI-specific data like token usage, estimated cost, and model quality measures in an APM dashboard, letting customers troubleshoot these applications in the same familiar workflow they’re already using.  

In addition to monitoring AI workloads, modern observability tooling uses AI itself to accelerate troubleshooting and help engineers get back to being productive. For example, the Splunk AppDynamics Agent leverages OpenTelemetry to enable customers to collect data once, then to use it either in Splunk AppDynamics or Splunk Observability Cloud (or both), giving customers the flexibility to use the observability offering that suits their needs without needing to rip-and-replace instrumentation.   

Once this data is collected, users can leverage natural language prompts through Splunk’s AI Assistants to more quickly identify issues that the data reveals. OpenTelemetry provides the fundamental data used to troubleshoot AI and the data needed for AI to help troubleshoot non-AI applications.   

A Mature, Future-proof Observability Practice   

For users who have embraced observability early, they are seeing how it drives key business outcomes in customer experience and product innovation and is a catalyst for AI adoption. As our research shows, OpenTelemetry is the foundation needed to build a robust observability practice. With our donation of the OpenTelemetry injector, organizations with diverse infrastructure can now adopt observability even easier and leverage it as a new enabler for success.  

For more information on how Splunk continues to support OpenTelemetry, visit here.  

For more information on the State of Observability report, visit here.