What Is SME-in-the-Loop in AI Evals?

Your LLM judge just evaluated a customer service response:

User: "Can I get a refund if I cancel after the trial ends?"

Bot: "Yes, you can request a refund within 30 days of your first payment. Just contact support and we'll process it right away!"

Judge verdict: PASS

Judge reasoning: "Response is helpful, addresses the user's question directly, and provides actionable next steps."

The SME's reaction: "That answer would get us sued. Our policy is 14 days, not 30. And 'right away' is a promise we can't keep."

Why do LLM judges need subject matter experts?

In regulated workflows, generic judges often plateau around "good enough" because they don't encode the domain rules that decide what's safe, compliant, or actually helpful in your specific context. In our deployments, the unlock from "mostly right" to "operationally safe" is usually domain criteria encoded by SMEs: people who've spent years learning what good looks like in your workflow.

Generic judges get you started. SME refinement gets you to production. A generic judge tops out near a ~70% accuracy ceiling; SME refinement takes it to 95% and production-grade evaluation.

SME review in the eval lifecycle

SME refinement is stage two of the eval engineering lifecycle: label a few hundred examples, sample the cases worth an expert's time, review the disagreements, extract the rule, re-measure. That loop turns a generic judge into one that encodes your domain.

How to build a ground truth dataset

Before any refinement work begins, you need a labeled dataset that serves as ground truth: the benchmark against which you'll measure whether your judges are actually getting better.

• Start with a representative sample. Pull 200-500 examples from production that reflect the actual distribution of queries.
• Include known failure modes. Add examples that target weaknesses you've identified, such as sarcasm, tool calls, subtle compliance violations.
• Label with documented reasoning. Every label should include the verdict and the reasoning. "Fail because the response recommends a specific investment without disclaimers" is useful; "Fail" alone is not.
• Measure inter-rater reliability. Have multiple labelers independently label at least 50 examples. Cohen's Kappa > 0.8 = strong agreement; < 0.6 = fix the rubric first.
• Refresh quarterly. Production distributions shift and new failure modes emerge.
• Minimum viable ground truth: 100 examples labeled by your lead domain expert with documented reasoning, stratified to include at least 20 examples of each major failure mode you're tracking. This is enough to detect meaningful accuracy changes (>5%) with reasonable confidence.

How do you measure a judge against ground truth?

Split your labeled data three ways:

• Train (~20%): examples you draw few-shots from
• Dev (~40%): examples you optimize your prompt against
• Test (~40%): final validation to catch overfitting

Don't report raw accuracy on imbalanced data. Use True Positive Rate (what % of real errors did we catch?) and True Negative Rate (what % of good responses did we correctly pass?). Aim for >90% on both. The refinement loop runs on top of that split:

• LLM judge. Few-shot evaluation.
• Judge vs. human. Measure TPR and TNR.
• TPR and TNR > 90%? If no: review disagreements with the SME, extract the rule, update the prompt, re-run the judge.
• Check the test set. If it holds, deploy. If it overfit, back into the loop.

Determining when SMEs need to review cases

You need SME eyes on actual failures, but SMEs are busy. Random sampling wastes their time on obvious cases; strategic sampling finds signal fast.

• Disagreement sampling (highest yield). Run two different judge prompts (or two different models) on the same cases. Where they disagree, something interesting is happening.
• Risk-based sampling (highest stakes). Regulated intents, irreversible actions, user PII — anything where a false negative has serious consequences.
• Novelty sampling (catches drift). Cluster recent production data by embedding similarity. Sample from clusters far from your training distribution.
• Production incident sampling (confirmed failures). User complaints, escalations, QA flags, and support tickets that mention the AI.
• Boundary sampling (high impact). Pull cases near your pass/fail threshold, where small prompt changes have outsized impact.
• The weekly review mix. Combine strategies: 10 disagreement cases, 10 risk-based cases, 10 novelty/incident cases. Thirty cases total, all worth expert attention.

Responsible areas in SME review programs

Teams get excited when they hit 80% agreement with human labelers. But it's the remaining 20% you need great SMEs and processes.

• Lead domain expert. One person with final decision authority on quality criteria. No committees, no escalation chains for routine disagreements. This person owns the rubric and approves all changes.
• Escalation path. When the lead SME is genuinely unsure, or the issue crosses into legal/compliance/product territory, escalate. Define this path in advance.
• Decision policy for ambiguity. In regulated domains, default to FAIL when uncertain. Better to over-flag and have humans review than to miss a violation.
• Change control for rubric updates. All criteria changes logged in the criteria library with date and rationale. Prompt versions tracked (v1.0, v1.1, v1.2). Metrics recorded before and after each change.

What do SMEs catch that LLM judges miss?

SME refinement surfaces two categories: implicit quality criteria that humans apply unconsciously, and systematic LLM judge biases.

• Confidence calibration. "The answer is X" vs. "probably X". The implicit rule: match confidence to evidence.
• Completeness. Correct but incomplete answers. The implicit rule: answer the real need, not the literal question.
• Tone mismatch. Right content, wrong delivery. The implicit rule: tone is part of correctness.
• Negative results. "I don't know" mislabeled as failure. The implicit rule: honest uncertainty is correct behavior.

When your SME consistently disagrees with your judge in a pattern, you've found a bias. Document it, add counter-examples, and track whether refinement reduces it. The earlier post in this series covers the full bias taxonomy with mitigations for each.

Understanding your agents at work is key to trusting AI systems. Learn about Splunk Agent Observability and get hands-on with the Splunk Observability Cloud Free Edition today

Common failure points for LLM judges

SME refinement improves judges, but some failure modes are structural. Know where judges reliably fail so you don't waste cycles trying to prompt-engineer around fundamental limitations.

LLMs miss behavioral and engagement signals

LLMs judge semantic correctness. They can't see what happens after the response. A support bot answer might be factually accurate, but if users immediately escalate to a human agent, something failed. Your SME sees the gap between what was said and what the user needed to hear.

Domain knowledge hides in unexpected places

An LLM doesn't understand that in healthcare, "unremarkable" means good, or that in insurance, "total loss" has a specific threshold. SMEs catch these because they've internalized thousands of domain-specific associations that might not have made it into the LLM's training data.

Sneaky overfitting when adding examples

Every time you add a rule or example to fix a specific failure, you push the model's attention away from the general problem. To avoid this: maintain a holdout set you don't peek at, and measure impact on the holdout.

Use judges for regression testing, not discovery

Trust your LLM judge to flag regressions on use cases your human labels cover. Don't trust it to discover new, unexpected failure modes.

FAQs about SME-in-the-loop