RAG Observability: How To Observe RAG Systems in Production

A RAG system that performs well in evaluation can still fail in production, and it usually fails in ways your test set never covered. Queries arrive that nobody anticipated. The corpus drifts as documents are added and removed. A model version changes underneath you. None of this shows up as an outage; instead, the system keeps returning fluent, confident answers, and the quality degrades quietly.

Observability is what makes that degradation visible. This guide covers what to trace, which evaluators to alert on, how to tell a retrieval problem from a generation problem in production, and — the part most observability writing skips — what to do once monitoring tells you something broke.

Monitoring vs. observability

Observability and monitoring are often mentioned together, but they answer different questions:

• Monitoring tracks predetermined metrics to tell you whether the system is healthy.
• Observability gives you insight into the inputs and outputs of a workflow along with every step in between, so you can ask questions you did not plan for.

In retrieval augmented generation (RAG), that distinction is concrete. Monitoring tells you latency rose. Observability lets you open a specific request, look at exactly which chunks the retriever returned and what scores they carried, see how the model used them, and confirm whether any guardrail fired. It is the difference between knowing something went wrong and knowing where.

Four must-have capabilities for reliable observability

Four capabilities make up a working observability setup.

Traceability

Traceability records the full path of a request. When a response looks wrong, you need to examine which chunks were retrieved, their scores, how the model used them, and whether guardrails were triggered. Frameworks like LangChain and CrewAI offer built-in tracing, and the structure that matters is the hierarchy: a workflow span as the parent, with retriever and LLM spans nested inside it.

Retrieval evaluation

Retrieval evaluation measures whether the correct information is reaching the generation layer. Chunk relevance and context relevance both run in production, not just during development, and changes in them are usually the first sign of a retrieval or context problem — well before anyone reports a bad answer.

Alerting

Alerting notifies you when performance deviates from normal. Typical alerts cover latency increases, error spikes, drops in retrieval scores and rising hallucination rates. Thresholds should come from baseline performance during a stable period rather than from a number that sounded reasonable at setup time.

Integrated tooling

RAG-focused observability platforms track hallucinations, retrieval scores, and safety evaluators using the same framework from development through deployment. That continuity matters more than it sounds: if your production evaluators differ from the ones you tuned against, you cannot compare a regression to the baseline that preceded it.

What to observe? Four areas for RAG observability

Trace and span structure

Observing the shape of execution is what makes everything else debuggable. A well-instrumented RAG request produces a workflow span containing a retriever span and one or more LLM spans, each with its own inputs, outputs, timing and token counts. Without that hierarchy you have a log line; with it you have a story you can follow.

Retrieved context

Observing what came back from the vector database is essential for assessing whether the information reaching the model was adequate. This means capturing the chunks themselves, not just their count: which chunks were selected, in what order, and whether any of them actually helped.

Model quality evaluators

These tell you how the model behaved with what it was given: whether it stayed grounded in the context, whether it covered the relevant information, and whether it stayed inside your safety and compliance boundaries.

System metrics

Resource utilization, latency and error rates tell you about the operational health of the deployment. They are the easiest to collect and the least informative on their own, which is why they are worth pairing with the three above rather than treating as the monitoring story.

Challenges with RAG: What actually goes wrong in production

• Evaluation doesn’t scale by hand: Manual sampling causes teams to miss rare but high-impact failures. (In contrast to automated evals, which enable full-volume analysis that captures the systemic impact of structural changes.)
• Hallucinations: AI fabrication creates hidden legal and financial liabilities because the system appears fluent and functional while generating errors that only surface long after the initial interaction.
• Toxicity and safety: Any AI interface accessible to users is vulnerable to adversarial probing, requiring robust safety controls to prevent agents from being manipulated into disseminating harmful or prohibited content.
• Failure tracing: Opaque pipelines force engineering teams into slow "change-and-hope" debugging cycles, whereas deep visibility into internals is essential for accurately pinpointing whether a failure originated in retrieval, prompting, or model behavior.

What to measure in production

Evaluators fall into seven categories. For a RAG system in production, three of them carry most of the weight: RAG evaluations, safety and compliance evaluators, and response quality evaluators.

RAG evaluators

RAG evaluators, split between retrieval quality and generation quality.

Safety and compliance evaluators

Safety and compliance evaluators. Each runs on inputs, outputs, or both.

Response quality evaluators

Response quality and expression evaluators.

Note on cost

Running LLM-as-a-judge evaluators across 100% of production traffic gets expensive fast, which is why most teams sample and then miss things. Luna evaluation models exists to remove that trade-off: purpose-built small language models, fine-tuned for evaluation, running at roughly $0.02 per million tokens against $2.50 for a frontier model, with average latency near 152 milliseconds rather than several seconds. Most evaluators have both a standard version and a Luna variant, so you can run the cheap one continuously and reserve the expensive one for spot checks.

System metrics

Three operational measures round out the picture, and one of them is usually done wrong.

• Latency: Track it as a distribution, not an average. As corpus size grows and indexes fragment, tail latency degrades first, so p50, p95 and p99 reveal problems long before the mean moves.
• Recall quality needs separate visibility, because declining retrieval accuracy never surfaces in latency metrics. Evaluate periodically against a held-out query set.
• Resource utilization and error rates: CPU, memory, disk and network, plus the frequency and type of errors encountered.

Product signals

User feedback mechanisms, such as thumbs up/down or star ratings, remain worth collecting. They are noisy and sparse, and they capture something no evaluator does: whether the answer was useful to the person who asked.

Guardrails: Catching problems before the user does

Observability tells you what happened. Guardrails stop some of it from happening at all: Input guardrails filter queries before processing; output guardrails validate responses before delivery.

On the output side, three checks cover most of the risk:

• Hallucination detection compares generated text against the retrieved context and flags anything that does not appear there.
• Format validation confirms the response matches structural requirements — that a request for JSON produced well-formed JSON.
• PII leakage detection scans for personal information the model may have generated or reconstructed, which matters even when source documents were anonymized.

These can run synchronously or asynchronously depending on your latency budget.

High-risk systems in finance or healthcare enforce strict blocking; internal productivity tools often run in logging-only mode to keep responses fast. Runtime guardrails takes this a step further by evaluating LLM and tool inputs and outputs during workflow execution and blocks harmful content, prompt injection, and PII leakage.

Finding the failures you didn’t know to look for

Every evaluator above answers a question you already knew to ask. The failures that hurt most in production are usually the ones nobody wrote a check for.

The traditional approach is to inspect traces manually until a pattern emerges, guess whether the issue lives in retrieval or generation, try to reproduce it locally where it behaves differently, and often wait for several user complaints before knowing there is a problem.

Signals inverts this, sweeping 100% of production traces to surface failure patterns that predefined evaluators and manual searches miss — security leaks, policy drift, cascading failures. Related failures are grouped, and each pattern gets a category and a priority from 1 to 10:

The prioritization question it asks is the right one: how harmful would it be if the user never discovered this issue? Once a pattern is worth tracking, it converts into an evaluator directly, which is how an unknown failure becomes a monitored one.

Setting meaningful alerts

The mistake most teams make is alerting on absolute thresholds picked at setup time. Establish a baseline during a stable period first, then alert on deviation from it. Four alerts cover most of what matters:

1. Latency increases, measured at p95 and p99 rather than the mean.
2. Error spikes, by frequency and by type.
3. Drops in retrieval scores, which usually precede quality complaints by days.
4. Rising hallucination rates, measured through context adherence.

The third one is the highest-value alert on the list and the one most often missing. Retrieval quality degrades gradually as your corpus grows and drifts, and it degrades silently — no error, no latency change, just answers that get a little worse each week.

Remediation: What to do when something breaks

Detection is only useful if you can act on it safely. Production RAG systems need mechanisms for rapid recovery and controlled change, and these are worth building before you need them.

Versioned components

Every major component has a version: embedding model, vector index, generation model, prompt template. Keeping at least one previous version of each means you can roll back immediately when a change reduces quality, instead of debugging under pressure.

Blue-green deployment

Maintain two environments. Route traffic to one while updating the other, validate thoroughly, then switch. If problems appear, switching back takes seconds rather than a redeploy.

A/B testing and gradual rollout

Route a portion of traffic to a candidate version and compare evaluator scores directly. This works for embedding models, retrieval strategies, reranking methods and prompts. Gradual rollout reduces risk further: start with a small percentage of traffic and expand only once the metrics hold.

The reason to instrument first is that all three of these require a number to compare. Without evaluators running on both arms, an A/B test is two opinions.

Compliance in production

RAG systems handling personal or regulated data have obligations under GDPR, CCPA, HIPAA and SOC 2, and the architecture has to support them. Four requirements come up repeatedly.

Right to erasure is the one that catches teams out. Deleting a user's data means removing it from raw documents, derived chunks, embeddings and indexes — every layer. A deletion that stops at the source system leaves the data live in the vector database, where it can surface in someone else's query months later. Automated workflows are the only reliable way to make sure all references go.

Audit trails should record the user identifier, the query, the retrieved documents and timestamps. Structured logging makes those traceable across services, and it is the same instrumentation your observability already produces.

Conclusion

Deploying a RAG system is the start of the work, not the end of it. The systems that stay reliable are the ones where someone can answer, at any moment, which chunks a given response was built from and whether the model stayed inside them.

That capability comes from a short list: trace every request with a real span hierarchy, run evaluators on live traffic rather than sampling, alert on deviation from a baseline instead of a guessed threshold, and keep versioned components so you can undo a bad change in seconds. Add Signals for the failures nobody thought to check, and guardrails for the ones you cannot afford to let through.

None of it is exotic. It is the same feedback loop that makes any production system improvable, applied to a component whose failures happen to be fluent and confident.

Splunk Agent Observability

Understanding your RAG system at work is key to trusting AI systems. Learn about Splunk Agent Observability and get hands-on with the Splunk Observability Cloud Free Edition today

FAQs about observability in RAG systems