How To Reduce Hallucinations in RAG Applications

When a RAG system hallucinates, the instinct is to reach for the prompt. Add a stricter instruction. Tell it to only use the provided context. Try a reasoning technique that worked in a paper.

Sometimes that helps. More often it does not, because the model was never the problem. Retrieval handed it the wrong chunks, or the right chunk ranked eleventh and never made it into the context window, or the context contained a factual error the model faithfully repeated. No amount of prompt engineering fixes a retrieval failure.

This article works through hallucinations in RAG from the other direction: what they actually are, the six patterns they take, where in the pipeline they originate, and the architectural changes that address each one. Prompting still has a role, and we cover it — but near the end, where it belongs.

What is a hallucination in RAG?

A hallucination in retrieval augmented generation (RAG) is information the model generated that is not present in the retrieved context. This definition is narrower and more useful than the general one, because it makes the failure measurable: you have the context, you have the response, and you can check whether every claim in the response is supported.

The concept of context adherence

This is what Context Adherence measures — closed-domain hallucination, on a scale from 0 to 1. A response scoring close to 1 contains only information given in the context. A response close to 0 is likely to contain facts that were never provided to the model.

It is worth being precise about what this excludes. A model that repeats an error present in its source documents is not hallucinating under this definition; it is faithfully grounded in bad context, which is a different failure with a different fix. Correctness catches that one. Keeping the two apart is what stops you from tuning a prompt when you should be cleaning a corpus.

Six common hallucination behaviors in RAG

Hallucinations in RAG are not one behavior, they are six, and each points somewhere different.

Noise robustness

Retrieved context contains a great deal of information, and the model has to work out which part is relevant. Noise robustness is its ability to extract what matters from a mixture of relevant and irrelevant documents. When it fails, the model answers from the wrong chunk — often one that is topically adjacent and semantically similar, which is exactly why it was retrieved.

Diagnose with chunk relevance, which labels each retrieved chunk as relevant or not relevant. If most chunks come back Not Relevant and the model still answered confidently, retrieval is feeding it noise.

Negative rejection

A RAG system has to know when it does not know. Negative rejection is whether the model declines to answer when none of the context is useful, rather than producing a plausible guess.

(This image shows a chat bot declining to answer when the retrieved documents are not relevant.)

This is the single highest-value behavior to test, because a system that guesses under uncertainty will guess in front of a customer. Context adherence is the measure.

Information integration

Complex questions usually require combining facts across documents. Information integration is whether the model can do that, or whether it answers from the first chunk that looked relevant and stops. The failure looks like a partially correct answer, which makes it harder to spot than an outright fabrication.

Completeness measures whether the response incorporated all the relevant information available in the context. If Context Adherence is precision, Completeness is recall — and you need both, because a response can be perfectly grounded and still useless if it covers a third of what was asked.

Counterfactual robustness

Some context documents contain errors. Counterfactual robustness is whether the model identifies known factual errors in what it retrieved and responds appropriately, rather than passing them through. Correctness identifies when responses reproduce errors from their sources.

Unclear queries

User queries are often unintentionally vague, particularly follow-up questions that depend on earlier turns. Faced with ambiguity, models tend to resolve it silently and answer the question they guessed. That guess is where the hallucination enters.

Incorrect citations

An incorrect citation is a reference to a source that does not exist, or information attributed to the wrong document. It is the most corrosive pattern in any domain where users verify sources, because the citation is what earned their trust in the first place. Every claim in the response must be traceable back to a retrieved document.

Where hallucinations enter the pipeline

Three of the six documented RAG failure modes produce hallucinations, and none of them happen at the prompt.

• Missing content: The query cannot be answered from the available documents at all. Your collection covers Q1 through Q3, the user asks about Q4, and instead of saying so the model fabricates numbers.
• Not in context: Relevant documents were retrieved but never reached the generator. Context consolidation removed them, token limits truncated them, or deduplication excluded them.
• Not extracted: The answer is present in the context, but the model did not use it. Noise obscured the relevant passage, or the model simply overlooked it.

Notice that only the third is a generation problem, and even that one is usually caused by too much retrieved context rather than by the model's reasoning. The other two are retrieval and context-management problems that a prompt cannot reach.

(This workflow image shows where each failure mode occurs in the RAG pipeline.)

How to fix hallucinations in RAG pipelines

Here are some solutions that actually make an impact.

Query rewriting

Vague queries produce vague retrieval. Query rewriting transforms incomplete or ambiguous input into a clear, specific request before it hits the vector database.

In conversational systems query rewriting matters even more, because a query like "What about business accounts?" is meaningless without the previous turn. History-based rewriting resolves the reference first, so retrieval searches for what the user meant rather than what they typed. This addresses unclear queries directly, and it is usually the cheapest intervention available.

Better retrieval and reranking

If the right chunk ranks eleventh, nothing downstream can save you. Two-stage retrieval fixes this:

1. Vector search returns a wide candidate set.
2. A reranker scores each candidate against the query with far more precision than similarity alone and reorders them.

A typical configuration retrieves the top 50 candidates and passes the top 5 reranked results to the model.

The effect on hallucination is measurable. Effective reranking pushes chunk-level relevance above 0.80 and improves Context Adherence, because the model is working from a smaller, denser context with less noise to get lost in. The cost is latency:

For most production systems, a cross-encoder at 200 to 400 milliseconds is the right trade. LLM-based listwise reranking is reserved for high-value queries or offline processing, where four to six seconds is affordable.

Context window management

More context is not better context. Retrieving 20 chunks when five would do increases cost, increases latency, and increases the chance the model latches onto the wrong one. Diversity mechanisms help here:

1. Maximal marginal relevance balances relevance against dissimilarity to what is already selected.
2. Autocut examines similarity scores and drops the tail.
3. Recursive retrieval fetches more only when the first pass was insufficient.

The signal to watch is chunk-level relevance. If most retrieved chunks come back Not Relevant while responses stay acceptable, you are retrieving more than you need — and you can cut cost and hallucination risk in the same change.

Output guardrails

Guardrails validate generated responses before they reach the user. Input guardrails filter queries; output guardrails filter responses.

This image shows a grounded, neutral response next to one an output guardrail should stop.

Hallucination detection compares generated text against the retrieved context and flags output containing information that does not appear there. Format validation confirms the response matches structural requirements. PII leakage detection scans for personal information the model may have generated or reconstructed even when source documents were anonymized.

These can run synchronously or asynchronously depending on your latency budget. High-risk systems enforce strict blocking; internal tools often run in logging-only mode to keep responses fast. Runtime guardrails takes this further, evaluating LLM and tool inputs and outputs during execution and blocking harmful content, prompt injection and PII leakage without requiring changes to agent code.

Measuring and diagnosing hallucinations

You cannot reduce what you do not measure, and "the model seems better" is not a measurement. Four evaluators cover the ground:

The order matters. Start with Context Relevance to see whether retrieval is working at all, then drill into Chunk Relevance to find which chunks let you down. If retrieval scores well and Context Adherence is still low, the problem really is generation — and that is the point at which prompting becomes the right tool.

Testing before you ship

Three tests catch most of what matters, and all three can be run this afternoon.

1. Noise filtering: Retrieve five documents where only one contains the answer, then ask a specific question. Does the model extract from the correct document, or blend in the irrelevant ones?
2. Negative rejection: Ask a question your documents cannot answer, such as a future quarter's revenue. Does the model admit it lacks the information, or fabricate?
3. Citation accuracy: Request an answer with citations, then verify each one references an actual retrieved document and quotes it correctly.

Fail any of these and you are generating ungrounded responses today. Beyond the spot checks, build a test dataset that covers all six patterns, keep roughly half real user queries and half synthetic, and add every production hallucination to it as a regression test. A dataset built this way grows 10 to 20 percent quarterly and gets more useful as it does.

Where prompting still helps

None of this makes prompting irrelevant. Once retrieval is sound and the context is clean, how you ask still affects what you get — and several research-backed techniques are worth knowing. We covered five of them in detail previously:

• Thread of Thought (ThoT) breaks down chaotic contexts and analyzes them piece by piece, selecting relevant information rather than reading passively. It suits long, messy context.
• Chain of Note (CoN) generates sequential reading notes for each retrieved document, evaluates relevance, then formulates the answer. It is the most directly useful of the five for noisy retrieval, since it makes the model assess each document explicitly.
• Chain of Verification (CoVe) drafts a response, plans verification questions, answers them independently, then revises. Effective but expensive, and harder to implement than the others.
• EmotionPrompt adds emotional stimuli to the prompt. Cheap to try, inconsistent in effect, and worth an experiment rather than a commitment.
• ExpertPrompting generates a tailored expert identity and has the model answer from it. Easy to implement and does not increase output tokens.

Two caveats. All five were evaluated on model generations that have since been superseded, so treat the reported gains as directional rather than as numbers you can bank. And every one of them costs input tokens, output tokens or both — which is a real trade when you are already paying for retrieved context.

Use them the way you would use any generation-side fix: after you have confirmed, with numbers, that retrieval is not the problem.

Conclusion

Hallucinations in RAG are mostly a plumbing problem wearing a reasoning problem's clothes. The model fabricates because the answer was not in front of it, or was buried in noise, or was contradicted by the chunk next to it. Fix the retrieval and most of the hallucination goes with it.

The order of operations that works: define hallucination narrowly enough to measure, instrument Context Adherence and Chunk Relevance so you know which half of the system is failing, fix retrieval and context management first, add output guardrails for what gets through, and reach for prompting last. It is less satisfying than a clever prompt, and it holds up in production.

FAQs about hallucinating RAG systems