How On-Premises AI Observability Works, and Why Regulated Enterprises Need It

In July 2025, an AI coding agent deleted a production database during an active code freeze, after being told repeatedly not to change anything without approval. It then reported that the deletion could not be rolled back. That was also wrong — the rollback worked. Replit's CEO called the incident unacceptable and shipped development and production separation in response.

The instructive part is not the deletion. It is that the agent took an action it had been told not to take and then misreported what it had done. You cannot manage that with logs the agent writes about itself. You need independent telemetry, and you need enforcement that sits between the agent and the systems it can damage.

For enterprises in financial services, telecommunications, healthcare and the public sector, that requirement collides with another one: the data those agents work on cannot leave a defined boundary. This is what pushes AI observability on-premises. Here is how it actually works.

Why AI telemetry is different

Conventional application observability moves metrics, traces and logs — request latencies, error codes, span durations. Little of it is sensitive on its own, which is why exporting it to a SaaS backend became normal practice.

Agent telemetry does not work that way. To evaluate whether an agent did the right thing, you have to capture what it was actually working with: the prompt, the retrieved documents, the arguments passed to each tool, the intermediate reasoning, and the final output. In a regulated environment that is precisely the data under control: the customer records, account details, clinical notes, subscriber information.

So, an AI observability pipeline is not a monitoring system that happens to touch sensitive data. It is a data-egress path, and it needs to be evaluated as one. That reframing is what most teams get wrong on their first attempt, because they scope the decision as an observability tooling choice rather than a data-flow change.

What AI regulations require

Three instruments shape this, and one of them creates a genuine bind. Under Article 12 of the EU AI Act, high-risk AI systems must technically allow automatic recording of events across their lifetime. Article 19 then requires providers to keep those automatically generated logs for a period appropriate to the intended purpose, and "of at least six months" unless other Union or national law provides otherwise.

Read those together with data-protection and residency obligations and the tension is explicit: you are required to retain six months of agent telemetry that frequently contains personal data, and separately constrained on where that data may be stored and who may process it. A SaaS observability backend in another jurisdiction satisfies the first requirement while complicating the second.

The NIST AI Risk Management Framework and its Generative AI Profile supply the control language most enterprise programs are written against, and the OWASP GenAI LLM Top 10 supplies the threat model — with prompt injection at the top and excessive agency now third, both of which are detected in exactly the telemetry described above.

Note the timing, because it is frequently misreported. The EU AI Act's high-risk obligations, including record-keeping, were deferred by the 2026 Digital Omnibus and now apply from December 2027 for standalone systems. The prohibited-practices and general-purpose model provisions were not deferred and are already in force. Teams building now have runway on record-keeping and none on the rest.

How on-premises AI observability works

A self-hosted AI observability stack has four moving parts. Most evaluations of these platforms focus on the first and get caught out by the second.

1. The telemetry layer

Instrumentation is the part that has standardized. OpenTelemetry now publishes semantic conventions for generative AI covering spans, metrics and events, including agent-specific spans for invoking an agent, calling a model and executing a tool, plus conventions for the Model Context Protocol. OpenTelemetry's own 2026 overview of the work describes the conventions as in use today and under active development, which is the right way to hold them: usable now, still moving.

OpenInference is the other convention in common use, built on OpenTelemetry with a span-kind taxonomy for LLM, agent, chain, tool and retriever operations. Because an OpenInference trace is a valid OTLP trace, the two coexist rather than compete. Preferring a platform that ingests both is a cheap way to avoid re-instrumenting later.

The practical piece is the collector. Running an OpenTelemetry Collector inside your boundary gives you a single enforcement point for redaction, sampling and routing before anything is written or forwarded. If you take one architectural decision from this piece, make it that one — a collector you control is what turns "we hope the SDK does not send too much" into a policy you can audit.

2. Where evaluation runs

This is the crux, and it is where on-premises AI observability differs from on-premises anything else.

Storing traces locally is straightforward. But a trace on its own tells you what happened, not whether it was correct. Turning telemetry into a judgment — was this answer grounded, did the agent pick the right tool, is this output safe — requires evaluation, and evaluation calls a model.

If that evaluator is a frontier model behind a SaaS API, then every prompt, retrieved document and tool output you wanted to evaluate leaves your boundary at evaluation time. Your traces never left. Your data did. Teams discover this after the architecture review, usually late.

There are three ways out, and they trade off differently:

• Redact before evaluating. Workable for narrow checks, but redaction removes exactly the context that grounding and correctness evaluations depend on, so quality drops in proportion to how well the redaction works.
• Self-host a frontier open-weight model as the judge. Keeps data local, but a large judge model per evaluation is expensive at production volume and slow enough to rule out inline enforcement.
• Use small, purpose-trained evaluation models. A model fine-tuned for one narrow judgment (Is this grounded? Is this a prompt injection?) can run on commodity GPUs inside your boundary, fast enough to sit inline. This is why small language model evaluators became the standard answer for on-premises deployments rather than a cost optimization.

3. Where enforcement runs

Observability tells you what happened. It does not stop anything. For the failure mode in the opening — an agent taking a destructive action it was told not to take — you need a control in the execution path, not a dashboard reviewed afterwards.

That means checks placed before or after a model call or a tool call, able to block the request, redirect the agent, or record and allow. And it means those checks run where the agent runs. An enforcement service reachable only across the internet adds a network round trip to every tool call and puts the payload outside the boundary in order to decide whether it should have left — which is self-defeating.

The design consequence is that evaluation and enforcement want the same infrastructure. If your evaluators already run locally and fast enough to be inline, enforcement is a configuration on top of them rather than a second system.

4. What you actually control in your own cluster

The reason to take on the operational burden of self-hosting is control over four specific things:

• Egress boundaries. Which endpoints the platform may reach, including whether it can reach public model APIs at all, or only gateways to models you manage.
• Identity and authorization. Federating the platform to your own identity provider, and scoping who can see traces containing regulated data.
• Retention. Setting retention to satisfy the six-month floor without exceeding what your data-protection obligations allow, in storage you administer.
• Auditability of the observability system itself. When infrastructure plays a role in regulatory evidence, you need records of actions taken inside it, not only the application traffic flowing through it.

The four deployment models for on-prem

"On-premises" covers a wider range than it appears. The distinction that matters is not where servers sit but where the boundary is drawn and who administers it.

Most regulated deployments land on the third row. Air-gapped is the right answer less often than it is requested, because it forecloses hosted frontier models entirely — which is usually a constraint on the agent, not just on its observability.

What to look for in a self-hosted platform

If you are evaluating options, these are the questions that separate platforms built for this from platforms that offer it as a checkbox.

• Can the evaluators run entirely on your compute, and on what hardware? A platform that self-hosts its UI and database but calls a hosted API to score anything has not solved the problem.
• Is enforcement available in the same deployment, inline with tool and model calls, or is it a separate hosted service?
• Which instrumentation standards does it ingest? OpenTelemetry and OpenInference support means your instrumentation survives a platform change.
• Can it be pointed exclusively at model gateways you manage, with public endpoints blocked?
• Does it federate to your identity provider and support role scoping fine enough to keep regulated traces away from people who should not see them?
• Are there audit records of actions taken inside the platform, not just the traffic passing through it?
• What is the upgrade path, and can you take it on your own schedule?

How Splunk Agent Observability deploys on-premises

Splunk Agent Observability can deploy on-premises, in a cluster you manage, so the evaluation and enforcement layers described above run inside your boundary rather than behind a hosted API. It supports the major model providers and the common agent frameworks.

The evaluation question is where it is most differentiated. Its Luna evaluation models are small language models fine-tuned for individual judgments, and the documentation confirms they can be self-hosted on-premises or on your own cloud infrastructure rather than only called as a service. Because each evaluator is trained for one narrow decision, it returns a verdict in roughly 150 milliseconds against several seconds for a frontier judge — which is what makes inline enforcement practical rather than theoretical. Published per-GPU latency figures cover the common datacenter accelerators, so capacity planning is not guesswork.

For enforcement, runtime AI guardrails attaches checks before or after an LLM or tool call and can deny a request, steer the agent, or observe and log. Controls take effect without redeploying the agent, and the product documentation states guardrails can run in your own VPC or fully air-gapped, with enforcement and logs staying under your control.

On the control surface: single sign-on federates to Entra ID, Okta, PingFederate, Google or a custom OIDC provider — and is available specifically when you run your own cluster rather than the hosted version, which is worth knowing when comparing tiers. Access control provides four system roles with per-project and per-dataset scoping. Ingest is OpenTelemetry and OpenInference, with documented integrations for LangGraph, CrewAI, Google ADK, the OpenAI Agents SDK, Microsoft Agent Framework, Pydantic AI and Strands.

One thing to size correctly during planning: self-hosted evaluators need GPU capacity in your environment, and the published latency tables are the input to that calculation. It is a real requirement, not an afterthought — but it is also the reason the data never has to leave.

Two adjacent capability pages cover the surrounding functionality: agent and model evaluation and the main product page.

Secure Architecture Makes the Difference

For regulated organizations, AI observability is not optional instrumentation. It is the evidence layer for a compliance obligation and the detection layer for a security control, and both have to work inside a boundary you administer.

The decision that determines whether that is achievable is not where you store traces. It is where evaluation and enforcement run. Get those inside the boundary and the rest of the architecture follows. Leave them outside and you have built a very well-instrumented data-egress path.

FAQs about on-premises AI observability