How to Set Up Evals with LLM-as-Judge

Using an LLM to evaluate your AI's outputs is deceptively simple to set up, and surprisingly hard to get right. This guide covers what LLM judges do well, where they systematically fail, and how to configure one for your own domain.

What is LLM-as-a-judge?

LLM-as-a-judge is the practice of using a large language model to evaluate your AI's outputs. The approach is deceptively simple. Write a prompt that defines what "good" means. Feed it your AI's outputs. Get judgments at scale. In theory, you've automated evaluation. In practice, most teams plateau at <70% accuracy and wonder what went wrong.

Why do LLM judges work?

LLM judges work because LLMs have internalized enormous amounts of human judgment. They've seen millions of examples of good writing, clear explanations, accurate summaries, and helpful responses. When you ask GPT or Claude whether a response is helpful, you're leveraging that internalized sense of quality.

This is why LLM judges can achieve 80%+ agreement with human evaluators on general tasks. Research from the MT-Bench paper showed that strong LLM judges match crowdsourced human preferences at roughly the same level that humans agree with each other.

What are the advantages of LLM judges?

1. They scale infinitely. They don't get tired, don't have bad days, and don't develop annotation fatigue after reviewing their 500th example.
2. They're consistent. Criteria are applied the same way across thousands of evaluations.
3. They're fast. Available 24/7, processing evaluations in seconds rather than days.
4. They're explainable. You can ask them to show their reasoning.
5. They appreciate over time. The judge you build today gets more accurate when the next generation of models arrives — point your pipeline at a better model and accuracy improves, with no retraining.

Why do generic LLM judges plateau?

Ask a frontier model to evaluate whether a customer service response is "helpful," and it will give you a reasonable answer. Ask it to evaluate whether that same response follows your company's specific escalation policy, uses approved terminology, and addresses the customer's underlying concern rather than just their stated question, and it will guess.

Generic judges plateau because they're optimizing for generic quality. They catch obvious failures: responses that are clearly wrong, incoherent, or off-topic. They miss subtle failures: responses that are technically correct but violate domain-specific requirements. Research from Notre Dame and Purdue quantified this: off-the-shelf LLM judges agreed with domain experts only 64-68% of the time on expert-knowledge tasks. That's the ceiling, not because the models can't judge better, but because they lack the domain context to do so. Legal review, customer support, and coding assistants each need custom evaluators built around their own criteria.

How do you build your first LLM judge?

Start with a single, specific failure mode. This constraint forces clarity about what you're actually trying to measure. Then compare what a weak prompt produces against a strong one.

Basic vs. refined eval prompts

A basic prompt will achieve roughly 60% agreement with human judgment:

It fails because "helpful" and "accurate" are vague. There's no context about the domain or use case, no examples of what pass/fail looks like, and no reasoning required before the verdict.

A refined prompt reaches 75-80% accuracy:

The difference is structural: a clear evaluator role with domain context, explicit numbered criteria instead of a vague "helpful," few-shot examples with reasoning, edge cases addressed rather than ignored, and reasoning-first output.

Prompt engineering principles for LLM judges

The quality of your judge depends almost entirely on the quality of your prompt. The same underlying model can achieve accuracy rates ranging from 60% to 95%, depending on the prompt used.

1. Be specific about the criteria. "Evaluate whether this response is helpful" gives the judge no framework. "Evaluate whether this response directly answers the user's question, provides actionable next steps, and avoids making assumptions about the user's technical expertise" gives it a concrete rubric.
2. Clarify vague or overloaded terms. "Uses appropriate tone" is ambiguous; "uses formal language, avoids slang, addresses user by name" is testable.
3. Use binary or categorical outputs. On a 1-5 scale, what's the difference between a 3 and a 4? PASS/FAIL needs no interpretation.
4. Include few-shot examples, especially for edge cases. Include examples for cases that caused disagreement. If your team argued about it, the judge needs it. Your examples are your specifications.
5. Add explicit decision rules for tricky situations. "If no citation → FAIL." "If partial answer without ask → FAIL." "If correct but wrong tone → FAIL."
6. Split compound criteria into separate evaluations. Don't evaluate "helpful AND accurate" in one pass — run one judge for each.
7. Require reasoning before judgment. Ask for the rationale first, then the verdict.

When humans disagree on labels, it's usually because the rubric is ambiguous. The same applies to LLM judges: unclear prompts produce inconsistent judgments. Tools like Autotune can automate this refinement loop once your criteria stabilize.

Conclusion

A generic judge gets you a baseline; a domain-specific one gets you to production. Start with one failure mode, write explicit criteria, add few-shot examples for the cases your team argued about, and measure agreement against human labels. Comparing your options across evaluator types is the fastest way to find where a judge fits your stack. Next in this series: panels of judges, the biases you have to mitigate, and what breaks in production

Learn more about Splunk Agent Observability and get hands-on with the Splunk Observability Cloud Free Edition today.

FAQs