Choosing an Embedding Model for RAG: Why Your Custom Evals Beat the Public Leaderboard

Retrieval looks fine in the demo. In production, the same question can return the wrong year's filing. Your embedding model decides which evidence reaches the generator. A poor choice can weaken answers, add latency, and force an expensive index rebuild. Public benchmarks screen candidates; only an eval on your corpus and queries finds the fit.

Defining embeddings: numerical representations in semantic space

An embedding is a numerical vector representing the semantic meaning of text, images, code, or other data, so related passages sit close together in a high-dimensional space. In a retrieval-augmented generation (RAG) pipeline, the embedding model decides which chunks reach the large language model (LLM), capping answer quality before generation starts.

The role of embeddings in question answering and tool retrieval

Embeddings play an important role in question answering, learning, and retrieval.

Question answering and conversational search

Vector search lets question-answering (QA) systems retrieve context by similarity despite different wording, rather than keyword overlap. In the OpenRAG study, 77% of Natural Questions queries had a findable answer in the corpus, yet an E5 retriever paired with Llama3-8B-Instruct answered only 54% of them. Tuning the retriever end-to-end for the generator added 4.0%.

Multi-turn dialogue changes what a query means, and later turns retrieve worse than early ones. The MTRAG benchmark included 110 conversations averaging 7.7 turns, and rewriting the query with conversation context lifted BGE-base 1.5 Recall@5 from 0.30 to 0.37.

In-context learning and tool fetching

You can embed the query and retrieve similar examples from a demonstration bank instead of using a fixed few-shot set. Purpose-trained selectors can improve retrieval for medical and legal text.

Agents can embed tool descriptions and retrieve options per request instead of fitting dozens or thousands of schemas in the prompt. On the ToolRet benchmark, the best off-the-shelf retriever reached only a normalized discounted cumulative gain at 10 (nDCG@10) of 33.83, every retriever stayed under 35% Completeness@10, and GPT-3.5's ToolBench-G1 pass rate dropped 11.40 points when tools came from bge-large retrieval.

Impact of embeddings on RAG pipeline performance

Vector dimension and performance evaluation

The Massive Text Embedding Benchmark (MTEB) leaderboard runs three tracks for shortlisting models:

• MTEB English v2 includes 41 English tasks.
• MTEB Multilingual v2 includes 131 tasks across more than 250 languages.
• RTEB tests production retrieval in legal, finance, code, and healthcare.

Rank uses a Borda count. The MTEB maintenance paper warns that top models may train on benchmark tasks while lower-ranked models generalize better, and broad rankings can diverge from specialized retrieval results.

Private vs. public embedding models

A hosted API gives you availability without hosting engineering but adds rate limits and usage charges. Open-weight models add deployment control but require GPU capacity and on-call attention. Choose the burden your team can absorb.

Cost considerations

Query costs cover per-token API prices or GPU bills. Indexing costs arrive when you switch models and re-embed the corpus, so store embeddings separately from your vector index for painless resets.

Storage scales linearly with dimension: the Pinecone cost formula, records × (ID size + metadata size + dimensions × 4 bytes), puts one million 1536-dimensional records with eight-byte IDs and 1,000-byte metadata at about 7.15GB, billed at $0.33/GB/month.

Quantization helps: in a quantization benchmark on 250 million vectors, binary embeddings shrank storage from 953.67GB to 29.80GB while keeping 96.45% of float32 nDCG@10 with rescoring, and int8 with a 4–5x rescore multiplier retained about 99%.

Search latency

Lower-dimensional vectors can improve throughput, but aggressive reduction can cost recall. Pick the smallest dimension your eval tolerates — cheaper vectors only pay off if recall holds.

Language support

An aggregate multilingual score doesn't guarantee consistent results in every language. Test the languages and regional vocabulary your users type, because an aggregate can hide weakness in your market.

Privacy concerns

Read the retention terms. Providers differ on model training, abuse-monitoring logs, retention, regional processing, and zero-retention options. Regulated data may require contractual agreements before sensitive records enter the pipeline.

Granularity of text

No single chunk size wins everywhere. A long-document study found 64–128 tokens suit concise, fact-based answers, while broad or technical questions did better with 512–1024 tokens, with the optimum shifting by model.

Types of embeddings

Dense embeddings

Dense embeddings are continuous, real-valued vectors with non-zero values in every element. Models like OpenAI's text-embedding-3 series, sentence transformers, and the Qwen3-Embedding family map a passage to one vector; approximate nearest neighbor search ranks documents by cosine similarity. They work when queries and documents express the same idea differently.

Sparse embeddings

Sparse embeddings hold mostly zeros and weight important terms, so they're strong on rare keywords and specialized vocabulary. Hybrid sparse-plus-dense retrieval earns its extra complexity only when your corpus is terminology-heavy, as in biomedical search, and your eval shows the lift; results vary by corpus and evaluation setup.

Multi-vector embeddings

Multi-vector models like ColBERTv2 encode queries and documents into one vector per token and score them with a lightweight late-interaction step. Precomputed document representations keep retrieval practical, while compression cuts index size and latency. The design also ranks portable document format (PDF) pages as images, helping when tables, layouts, or diagrams carry meaning that text extraction loses.

Long-context embeddings

Current models accept far longer inputs, though original models can struggle as input grows while context-extended configurations do better. Adding document-specific context to each chunk before embedding can reduce retrieval failures, especially with lexical retrieval and reranking.

Variable-dimension embeddings

Matryoshka Representation Learning (MRL) trains one embedding so its leading dimensions form usable smaller embeddings, and one model then supports multiple storage and latency targets. Several commercial and open-weight families support truncation, though acceptable truncation depends on the model. One recent finding tempers the story: plain truncation of non-MRL models can remain competitive until embeddings are reduced very aggressively.

Code embeddings

Code embedding models train on docstring and code pairs, so they match a query to a function by intent rather than shared tokens. CoIR is a standard test across code retrieval datasets and languages. Newer specialized models have superseded earlier baselines; Jina and Voyage both ship code-focused families.

How to measure embedding performance

MTEB retrieval tasks contain a corpus, queries, and a mapping from queries to relevant documents. The model embeds everything, cosine similarity ranks the corpus, and nDCG@10 scores the results. Many are repurposed QA sets with strong lexical overlap, which flatters keyword-matching models.

So, you evaluate on your own data: usually raw text, no relevance labels. In Splunk Agent Observability, three evaluators cover the retrieval side without needing a labelled gold set.

• Chunk Relevance is binary and chunk-level: each retrieved chunk is either Relevant or Not Relevant. If nothing comes back Relevant for a query, retrieval failed for that query. It is deliberately lenient — partial relevance counts.
• Context Relevance asks the same question of the retrieved context as a whole: does it contain enough information to answer the query? Start here, then drill into Chunk Relevance to find which chunks let you down.
• Context Precision adds position: what share of the retrieved context was relevant, weighted by where each chunk ranked. Precision @ K does the same at a specific cut-off.

Note that Chunk Relevance is computed by prompting an LLM, so it costs additional calls. Budget for it on the sweep rather than on every production query, or run it on a held-out sample.

Choosing the right embedding model

Here's how Chunk Relevance compares encoders on 10-K annual financial reports in a simple RAG system.

Data preparation

We parsed four years of Nvidia 10-K reports with PyPDF, skipping advanced chunking, to produce 701 large text chunks.

We sampled 100 chunks across the reports and prompted a current frontier model for one short question each. Any capable model works here — the questions are the artifact, not the generator.

QA chain

The chain uses LangChain with a Pinecone serverless index as retriever, k=3. The model generates under the system prompt, "Answer the question based only on the provided context."

RAG evaluation metrics

For every run, Splunk Agent Observability computed Chunk Relevance, Context Relevance, Context Adherence and Completeness. Luna evaluation models are purpose-built for production evals, which is what makes it affordable to score all 100 questions on every configuration rather than sampling.

Workflow

One function takes the experiment parameters and, for each configuration, it:

1. Loads the embedding model.
2. Creates a fresh index at the target dimension.
3. Embeds the 701 chunks and upserts them.
4. Tags the run with the encoder name, LLM, dimension, and top k.
5. Runs the 100 questions and logs each answer, retrieved chunks, and evaluator scores.

We pass the function to the experiment runner for repeatable RAG and multi-step workflow tests.

Sweep

We compared all-MiniLM-L6-v2 with text-embedding-3-small and text-embedding-3-large, both truncated to 384 dimensions to match all-MiniLM-L6-v2. Switching to text-embedding-3-small raised the share of chunks scored Relevant by roughly 7 points — the OpenAI encoder surfaced more chunks that actually bore on the question. Small and large were nearly identical at 384, so the cheaper model won. Newer compact models like Qwen3-Embedding and Jina's small families are the better candidates to test next.

Filtering to the samples where no chunk came back Relevant surfaces a pattern: income-tax questions pinned to a specific year. The retrieved chunks discuss income tax, but none cover the year asked about. That is a granularity and metadata problem rather than an encoder problem — which is exactly the distinction a sweep is for.

Common embedding failures and how to fix them

The income-tax case above is one instance of a pattern. Embeddings fail in five predictable ways, and knowing which one you're looking at decides where the fix goes — because most of them are not fixed by swapping the encoder.

Domain vocabulary mismatch

Your documents use terminology the model never saw in training. Medical abbreviations, legal citations and company-specific acronyms produce poor embeddings because the model has no context for those terms. Fix: fine-tune on a sample of your domain documents, or add sparse embeddings alongside the dense ones so term-level matching can carry the rare vocabulary.

Temporal reference failures

Embeddings capture topical similarity but miss time-specific qualifiers. A query for "2024 Q3 revenue" returns documents about revenue in general, not that quarter. This is the failure the sweep above surfaced. Fix: extract temporal entities explicitly and use them as metadata filters alongside the embedding search, rather than expecting the vector to carry the date.

Negation and qualification errors

Embeddings struggle with logical operators. "Python code without external dependencies" retrieves all Python code, because the constraint carries almost no weight in the vector. Fix: LLM-based reranking, which can reason about constraints the embedding missed.

Length mismatch

A five-word query embedding sits far from a thousand-word document embedding even when the document answers the query thoroughly. Fix: embed documents at multiple granularities — paragraph, section, full document — and retrieve across all levels. Query expansion to generate longer variants also helps.

Ambiguous query interpretation

Vague queries produce embeddings that sit between concept clusters. "Apple performance" could be fruit nutrition or stock results, and the vector lands between them. Fix: query expansion to generate multiple interpretations, or a minimum keyword match through hybrid search to disambiguate.

Worth running on your own system: take your last ten failed retrievals, categorize each against these five, and count. If six of ten are temporal, metadata filtering will beat fine-tuning by a wide margin. If most are vocabulary, the reverse holds. The distribution tells you where to spend.

Building a reliable embedding selection process

Leaderboards narrow the list; custom evals decide whether a model meets your quality, latency, cost, language, and privacy requirements. A sweep separates encoder failures from chunking and metadata problems.

When the wrong chunk reaches the generator, the agent fails in front of a user. Splunk is the intelligence layer for trusted agentic operations, and Splunk Agent Observability applies the visibility and evaluation discipline you trust for your services to your agents:

• Evaluators: Measure Chunk Relevance, Context Relevance, Context Precision, Context Adherence and Completeness across embedding experiments.
• Experiment comparison: Compare configurations side by side across quality, latency, and token usage.
• Luna evaluation models: Run purpose-built production evals without relying on expensive general-purpose judges.
• Signals: Surface recurring failure patterns before they become widespread incidents.

Read The Agentic Shift to see how Splunk extends observability to the agents your retrieval pipeline serves.

FAQs about embedding models for RAG pipelines