What Is BERTScore and How Does It Work for NLP Evaluation?

Your model answered correctly in different words, and your eval scored it poorly. Traditional n-gram metrics such as bilingual evaluation understudy (BLEU) and recall-oriented understudy for gisting evaluation (ROUGE) reward exact word overlap, and that approach frequently conflicts with human judgment on large language model (LLM) outputs across machine translation, text summarization, and open-ended generation.

BERTScore addresses this limitation with contextual embeddings from transformer models such as Bidirectional Encoder Representations from Transformers (BERT), RoBERTa, and DeBERTa. It scores semantic similarity in context rather than relying only on matching words.

What is BERTScore?

BERTScore provides a semantic similarity metric for natural language processing (NLP) that goes beyond surface-level word overlap. It computes token-level cosine similarities between contextual embeddings, then aggregates them into precision, recall, and F1 scores.

Say you're evaluating an e-commerce support response. "The cat sits on the mat" and "A feline rests upon a rug" share no words but convey similar meaning. BERTScore can recognize that relationship because it compares context-dependent representations rather than exact strings.

The metric supports multiple pre-trained models. The official repository recommends microsoft/deberta-xlarge-mnli for its strongest human correlation, or microsoft/deberta-large-mnli as a faster alternative.

How BERTScore works step by step

BERTScore turns text similarity into a three-step process:

1. Embed tokens in context.
2. Match them with cosine similarity.
3. Optional: Reweight and rescale the results.

Here's what happens at each stage.

Generating contextual token embeddings

When you're tracing why a semantically correct output scored poorly, BERTScore tokenizes your candidate and reference texts, then passes both through a pre-trained transformer. Each token receives an embedding shaped by the words around it, so "bank" gets different representations in "river bank" and "bank account." That context dependence lets the metric recognize paraphrases and synonyms that exact matching misses.

You specify the transformer through model_type and select a layer with num_layers. Both choices affect score distributions, inference time, and graphics processing unit (GPU) memory use. Larger models may improve correlation for some datasets, but they raise evaluation cost and latency.

Matching tokens with cosine similarity

BERTScore computes pairwise cosine similarities between every candidate token and every reference token, producing a matrix in which each element represents the contextual similarity of one token pair. Greedy matching then selects the strongest available relationship for each token, as defined in the original BERTScore paper.

Precision starts from the candidate: for each candidate token, BERTScore finds the most similar reference token and averages those maximums. Recall reverses direction, finding the best candidate match for every reference token. The F1 score is their harmonic mean, balancing added content against missing content.

Consider a developer assistant that gives a correct explanation but includes an unrelated troubleshooting step. Recall may stay high because the reference content appears, while precision falls because the extra material matches poorly. Reviewing all three values separates omission from unnecessary generation.

Applying IDF weighting and baseline rescaling

Two optional settings change how BERTScore behaves, and both are disabled by default. IDF weighting, set with idf=True, emphasizes rare, informative tokens over common words. The repository cautions that IDF estimates become unreliable when your reference collection is too small, so calculate them from a representative evaluation set.

Baseline rescaling addresses interpretability. Raw values often cluster between 0.85 and 0.95 with large RoBERTa models, which makes modest improvements hard to explain. Setting rescale_with_baseline=True maps results to a more readable scale without changing correlation rankings, according to the official documentation.

BERTScore vs. BLEU vs. ROUGE

Choose the metric based on which failure matters most. BLEU and ROUGE ask whether you used the same words. BERTScore asks whether you meant the same thing.

Comparing alignment with human judgment

BERTScore aligns more closely with human evaluation than lexical metrics in some domains, but the advantage changes by task. The ExPerT study reported 59% alignment with human judgments on personalized long-form generation, while BLEU reached 47% and ROUGE-L reached 50%. LLM evaluators GEMBA and G-Eval reached 69%, and ExPerT reached 74%.

Healthcare results are less encouraging. An October 2025 Digital Medicine analysis found weak, inconsistent correlations across clinical completeness and correctness tasks. Semantic similarity can't establish medical accuracy by itself.

Translation is more balanced. The 2025 machine translation shared task reported system-level accuracy of 0.78 for BERTScore and 0.79 for BLEU; segment-level accuracy reached 0.59 and 0.58, respectively. Treat automated metrics as screening tools, then use human review wherever an incorrect output creates material customer, financial, or clinical risk.

Choosing the right metric for your task

Use BERTScore when semantic accuracy matters more than exact wording: paraphrases, creative text, open-ended answers, and abstractive summaries with multiple valid phrasings. The trade-off is transformer inference — which adds latency and usually benefits from GPU acceleration.

BLEU or ROUGE remains useful when speed, reproducibility, or established benchmarking matters most. They fit quick regression tests because they require no neural model.

Similarity scores can stay high even when a model changes a date, diagnosis, price, or application programming interface (API) parameter, so factual grounding needs its own checks.

Best practices for implementing BERTScore in AI evals

Getting reliable numbers out of BERTScore takes more than a one-line function call. These practices help you set up pipelines, keep results comparable across runs, and catch regressions before they reach production.

Setting up efficient evaluation pipelines

When you're wiring evals into a production pipeline, install bert-score, transformers, and torch, then run the model on a GPU when possible. The repository notes that BERTScore is computationally intensive. Start with a simple implementation:

For repeated evals, use the BERTScorer object instead of the functional API, since it caches the transformer while repeated score calls reload the model. The default batch_size is 64; lower it when GPU memory is constrained.

Configuring comparable and reproducible evals

Cache model weights, align batch sizes with available memory, apply consistent text preprocessing, and avoid switching transformer models inside one benchmark, because each model produces a different score distribution.

Set rescale_with_baseline=True when you need an interpretable dashboard. If rare terms carry important meaning, precompute an idf_dict from a representative corpus and reuse it across batches.

Store the configuration hash the library emits beside the dataset and application version, so your team can attribute a score move to a prompt update, model release, reference revision, or infrastructure change.

Monitoring scores and maintaining baselines

An average hides the batch where your agent fell apart. Track score distributions rather than only averages, and review the lowest-scoring samples after each significant model or prompt change. Establish logging around your scorer so failed batches don't silently disappear:

Pin your library, transformer, and model versions instead of auto-updating them, and run dependency upgrades as controlled experiments against your validation set. Alert on changes in score distribution, missing batches, and processing latency so your team detects evaluation drift before it becomes a misleading quality signal.

Common BERTScore use cases

BERTScore is most useful when several outputs can express the same valid meaning. Validate it within each domain and pair it with metrics for factuality, safety, and task completion.

Shared tasks and benchmarks

BERTScore continues to appear in academic benchmarks. The SemEval benchmark includes it in a composite metric alongside ROUGE, and recent translation assessment research uses it as an embedding-based baseline for newer learned evaluators.

Text summarization

BERTScore helps when semantic fidelity matters more than exact overlap. A SaaS support summary might say "The customer cannot access billing settings" while the reference says "The account owner is locked out of payment controls." ROUGE may penalize the wording difference; BERTScore can recognize the shared meaning. The clinical results above show why healthcare summaries still require expert review.

Content and workflow quality

BERTScore can compare human-authored and generated task flows, support replies, product descriptions, and technical explanations.

Building reliable production evals with BERTScore

BERTScore gives you a stronger view of whether semantic quality changed than exact-overlap metrics, especially for paraphrases, summaries, and open-ended responses. How much that view is worth still depends on your model choice, representative references, a frozen configuration, and periodic checks against human judgment.

Splunk is the intelligence layer for trusted agentic operations across the enterprise, extending observability to the AI agents now running on it. Splunk Agent Observability turns isolated BERTScore results into repeatable production-quality decisions:

• Metrics: Combine built-in and custom metrics beyond semantic similarity.
• Custom metrics: Add deterministic BERTScore logic to shared regression workflows.
• Splunk Agent Observability's Luna evaluation models: Evaluate production traffic with purpose-built models in under 200ms and at up to 96% lower cost than frontier LLM-as-judge.

FAQs about BERTScore for NLP