Optimizing RAG Retrieval: How to Select the Right Reranking Model

Your support assistant gives a wrong answer even though the correct policy sits in your knowledge base. The problem may be retrieval order — rather than missing data — and in a RAG system the reranker fixes that order.

When you're tuning retrieval for a production retrieval-augmented generation (RAG) system, these reranking decisions matter most.

Defining rerankers: the second-pass filter in retrieval pipelines

A reranker is the second-pass filter in an information retrieval (IR) system. It reorders documents from the first-stage retriever (semantic, keyword, or hybrid search) so the most relevant land at the top of the context window. Rerankers trade efficiency for effectiveness, applying heavier matching than a vector inner product across the candidates the retriever provides.

This image shows the two-stage retrieval process: the vector database returns a wide candidate pool, the reranker narrows it before generation.

Why reranking is necessary for production RAG pipelines

Unrelated docs may reach the context. A distracting passage study found one hard distractor cut accuracy by 6–11 points, depending on the LLM; Llama-3.2-3B dropped from 82.6 with only the relevant passage to 71.5 with one distractor. Rerankers push relevant chunks up and distractors out, lowering hallucination risk and token spend. In the DynamicRAG ablation, the full system averaged 47.1 exact match versus 34.5 without reranking, a 12.6-point gap.

Bridging the semantic gap: why embeddings require reranker support

Embeddings often miss contrast, like "I love apples" versus "I used to love apples." The NevIR benchmark, reproduced in a SIGIR 2025 study, tests pairs differing only in negation: most bi-encoders score 5–12% pairwise accuracy, the cross-encoder jina-reranker-v2-base-multilingual reaches 65.2%, and the listwise LLM reranker o3-mini reaches 77.3%.

Training data limits generalization to unseen documents and queries. SFR-Embedding-Mistral scores 59.0 on a general embedding benchmark but 18.3 on the reasoning-heavy BRIGHT benchmark, measured by normalized discounted cumulative gain at 10 (nDCG@10). Benchmark contamination may explain the gap: a 2025 analysis argues that top embedding models train on zero-shot benchmark training sets and arguably test corpora, weakening those benchmarks as zero-shot signals.

Dimensionality constraints

Embeddings compress every document into a fixed number of dimensions, and the LIMIT paper shows this caps how many distinct top-k result sets a model can ever return: the maximum corpus size grows only polynomially with dimension, while the number of possible top-k combinations grows combinatorially. Even under idealized conditions, the authors estimate the breaking point at roughly four million documents for 1024-dimensional embeddings, and real models hit the wall far sooner.

Reranker mechanics: Cross-encoders versus late-interaction models

Cross-encoders and late-interaction models like ColBERT score contextualized units instead of compressing each document into one vector.

Comparing reranker architectures: performance, cost, and latency trade-offs

Cross-encoder architectures: pairing queries with passages for relevance

A cross-encoder scores query and passage together, so each candidate must be paired with the query and run through the model. Documents can't be embedded independently.

A cross-encoder processes the query and document together rather than embedding each independently.

Here is a BGE reranker:

Selecting cross-encoder options

The BGE v2 and v2.5 models supersede bge-reranker-base and bge-reranker-large. Options include the efficient, multilingual bge-reranker-v2-m3 (0.6 billion parameters) and performance-focused bge-reranker-v2-gemma and bge-reranker-v2.5-gemma2-lightweight. Mixedbread's mxbai-rerank-v2 is an Apache 2.0 alternative with eight-thousand-token context. Choose a cross-encoder when relevance matters more than minimum latency and costs must remain predictable.

Late-interaction models: scaling retrieval to large document corpora

Multi-vector models like ColBERT encode query and document text independently into token-level vectors, then interact at scoring time. Pre-computed document representations reduce compute for large collections, and ColBERTv2 adds residual compression that shrinks the index while maintaining quality.

ColBERT computes token-level interactions between query and document at scoring time.

Late-interaction scoring looks like this:

Matrix multiplication scores every query term against every document term, max-pooling keeps the best match per query term, and the sum ranks the documents.

Multi-vector options for retrieval

GTE-ModernColBERT-v1 is a compact Apache 2.0 model with long usable context, and jina-colbert-v2 supports long context across many languages, reducing reliance on chunking. Use this class when your corpus is too large to cross-encode every candidate.

LLM reranking methods: Pointwise, listwise, and pairwise methods

Fine-tuning gets expensive past 10 billion parameters, so recent work prompts LLMs to rerank zero-shot:

• Pointwise methods: Score one query-document pair with a judgment or query-likelihood score.
• Listwise methods: Prompt with a query and document list, then return reranked identifiers.
• Pairwise methods: Compare two documents per query and aggregate preferences.
• Setwise methods: Compare small groups to cut inference calls.

Listwise reranking processes all candidate documents together and returns a reordered list.

Candidate sets rarely fit one prompt, so RankGPT-style methods slide a window back to front; smaller steps bring modest gains but increase inference time.

LLM reranking trade-offs

Pairwise aggregation ranges from AllPairs at O(N²) to sorting variants at O(N log N) or O(N). Current candidates include GPT-5 and GPT-5-mini, Claude Sonnet 4.5 and Claude Haiku 4.5, and Gemini 2.5 Pro and Gemini 2.5 Flash. Top-100 frontier reranking is expensive, so reserve it for reasoning gains worth the latency and spend.

Supervised LLM rerankers: Fine-tuning for ranking awareness

MS MARCO fine-tuning gives base models ranking awareness; RankT5 and RankZephyr now serve as dated baselines. Alibaba's Qwen3-Reranker family is the open-weight standard: Apache 2.0, 32,000-token context, more than 100 languages, and pointwise yes/no classifiers in sizes of 0.6 billion, four billion, and eight billion parameters. The 0.6 billion-parameter model scores 65.80 on a retrieval benchmark against 57.03 for the same-sized bge-reranker-v2-m3, and the four-billion-parameter model reaches 69.76.

Rank1 research shows a seven-billion-parameter reasoning model averaging 27.5 nDCG@10 on BRIGHT against 17.5 without reasoning, though another 2025 study found a non-reasoning variant won by roughly three points on BRIGHT across model sizes from 1.5 billion to seven billion parameters. Test on your query mix before paying for chain-of-thought.

Private reranking APIs: Private reranking APIs vs. open-source models

Cohere recommends rerank-v4.0-pro and rerank-v4.0-fast for long-context multilingual reranking; long documents become billable chunks. Voyage offers rerank-2.5 and rerank-2.5-lite at other price points. Jina's listwise jina-reranker-v3.5 scores candidates in one long-context pass, though self-hosting from v2 onward falls under CC-BY-NC 4.0. Mixedbread provides a usage-based hosted endpoint for its Apache 2.0 mxbai-rerank-v2 weights.

One practical caveat before you write any code: the LangChain integration used later in this article still pins Cohere's v3.5 generation. If you are building on LangChain today, evaluate v4.0 through Cohere's own SDK and treat the pinned version in the sample below as a framework constraint rather than a recommendation.

How to select a reranker

Evaluate relevance, latency, context handling, and domain shift together:

• Relevance improvement: Measure retrieval with nDCG and generation with grounding metrics. A 2025 scientific-RAG benchmark found cases where Recall@10 rose after reranking while answer quality fell below the retrieval-only baseline.
• Latency considerations: Measure added latency. Hardware, candidate count, sequence length, and batching all affect results.
• Contextual understanding: Context windows range from hundreds of tokens in older rerankers to tens of thousands in newer models and hosted APIs, allowing whole-document reranking instead of chunks.
• Generalization ability: On BRIGHT, MS MARCO-trained cross-encoders reduced retrieval quality while GPT-4 reranking improved BM25 by 3.1 points. If queries resemble standard retrieval benchmarks, use a cross-encoder. If they resemble BRIGHT, budget for an LLM or reasoning reranker.

Latency is the constraint that most often decides this, so it is worth budgeting with real numbers rather than adjectives. As a starting point for a typical production pipeline reranking 20 to 50 candidates:

Typical latency added by reranker class. Measure on your own hardware — published figures for domain-specific corpora run considerably higher.

Those figures assume short passages and batched inference. Reported latencies from published research vary widely with the workload: one study of 1,200 SEC filings measured 2.02 seconds average for cross-encoder reranking on long financial documents, an order of magnitude above the typical range. Long documents, unbatched calls and cold starts all move this number.

Practical insights from current reranker research

Research supports three practical recommendations:

Choose models by query difficulty: Cross-encoders remain competitive on standard benchmarks, while LLM and reasoning rerankers gain ground on BRIGHT-style queries.

Evaluate end-to-end outcomes: An EACL 2026 paper notes traditional ranking metrics don't perfectly model how generators consume context, so compare retrieval gains with groundedness, completeness, and final answer quality.

Tune candidate depth: Optimal depth research shows many queries reach a point beyond which effectiveness stagnates; oracle depth selection improved MS MARCO Dev by over 7% while cutting average depth by a factor of five. Large-pool research found Recall@10 fell below retrieval-only in 53.3% of academic-dataset experiments.

Common reranker failures and how to fix them

Rerankers fail in predictable patterns. Recognizing them is faster than re-running a model sweep, because most of these are fixed by changing how you feed the reranker rather than by changing the reranker.

Phantom hits

The reranker assigns high scores to irrelevant documents that contain query terms in misleading contexts. A query about "Python programming" ranks a document about "Python snakes in programming conservation efforts" highly because both terms appear, even though the semantic relationship is wrong.

Fix: combine cross-encoder scores with keyword matching so documents have to contain query terms in appropriate contexts, not just anywhere in the text. Thresholding also helps — reject anything scoring below 0.3 even if it lands in the top results.

Context truncation

Documents longer than the reranker's context limit get truncated, and everything past the cutoff never influences the ranking. If your reranker handles 512 tokens and your average document runs 800, you are scoring the first two thirds of each candidate and guessing at the rest.

Fix: chunk long documents before reranking and treat each chunk as its own candidate, then aggregate scores back to the parent document. Or move to a longer-context model — bge-reranker-v2-m3, mxbai-rerank-v2 at eight thousand tokens, or one of the hosted APIs at 32,000.

Length mismatch

Short queries paired with long documents, or long queries against short snippets, produce inconsistent scores. Rerankers trained on balanced query-document pairs struggle when the ratio drifts far from what they saw in training.

Fix: expand short queries with terms from the initial retrieval results before reranking, or extract representative sentences from long documents rather than passing them whole.

Domain mismatch

Rerankers trained on web text underperform on specialized vocabulary. A model trained on Wikipedia and news will struggle with medical terminology, legal citations or technical specifications because it never learned those patterns. This is the same effect the BRIGHT results above describe, seen from the practitioner's side.

Fix: fine-tune on domain-specific query-document pairs — even 1,000 to 5,000 examples help substantially. Where fine-tuning isn't feasible, start from a model pre-trained on relevant material rather than a general one.

Query ambiguity

Vague queries produce unreliable rankings. When a query could mean several things, the reranker optimizes for one interpretation while the user intended another, and the ranking looks confident either way.

Fix: generate multiple query interpretations before reranking and score candidates against each, then either surface the top results per interpretation or use retrieval patterns to infer the most likely one.

Evaluating your reranker: A framework

Say you're evaluating a question-answering system on NVIDIA's 10-K filings with a Cohere reranker. CohereRerank lives in langchain-cohere with a mandatory model argument, ContextualCompressionRetriever is in langchain-classic, and the Pinecone package is pinecone rather than pinecone-client.

Here emb_k sets how many documents the embedding model returns and rerank_k sets how many the reranker keeps. Sweep both depths, starting at emb_k=10 and rerank_k=3, and run each configuration against the same questions with and without the reranker.

Within Splunk Agent Observability, for example, evaluators tell you whether the reranker worked.

• Chunk Relevance labels each retrieved chunk Relevant or Not Relevant, so it shows directly whether reranking raised the share of useful chunks reaching the generator.
• Context Adherence measures closed-domain hallucination — whether the response stayed inside what was retrieved.
• Context Precision adds the positional dimension, weighting relevant chunks by where they ranked, which is exactly what a reranker is supposed to move.

Read them in order. Low Context Relevance points at the retriever, and reranker tuning belongs in that branch. High Context Relevance with low Context Adherence points at generation instead, and no amount of reranking will fix it.

These evaluators run on Luna evaluation models at $0.02 per million tokens, 152ms average latency, and 0.95 F1, allowing you to score every question instead of sampling. Swap in Voyage, Jina, or a self-hosted BGE model and rerun the sweep.

Building a reliable reranking strategy

Match model type to query distribution, latency budget, context lengths, and quality targets: cross-encoders and purpose-built APIs for semantic retrieval, late-interaction models for large collections, and LLM or reasoning rerankers for hard out-of-domain queries. As RAG becomes agentic, Splunk is the intelligence layer for trusted agentic operations, connecting reranker experiments to production controls:

• Evaluators: Compare Chunk Relevance, Context Relevance, Context Precision, Context Adherence and Completeness, alongside your own criteria.
• Luna evaluation models: Run production-scale evals at sub-200ms latency and lower cost.
• Agent Observability: Trace retrieval and generation in production agents.
• Signals: Surface recurring failure patterns without reading every trace.

Splunk's mission is to keep digital systems secure and reliable, including the agents built on your retrieval stack. Read The Agentic Shift to see how Splunk extends observability to those agents.