SLM-as-Judge: How to Build and Deploy an SLM Judge

LLM judges hit a scaling wall: cost and latency force sampling, and sampling misses failures. SLMs break through by trading generalization for efficiency. The order matters — refine with LLMs until criteria stabilize, accumulate labeled data through SME cycles, train once volume or latency exceeds what an LLM judge can carry, then keep the hybrid.

Deciding when to use SLM judges is the easy part. Deploying it can get tricky—and this tutorial has you covered: where to start, how to build and serve an SLM judge, and common pitfalls to avoid.

Which SLM judges already exist?

The claim that fine-tuned SLMs can match or beat frontier LLMs on narrow evaluation tasks isn't theoretical. Production systems prove it daily, and they are what you can start from.

Llama Guard

Llama Guard is Meta's approach to safety classification: fine-tune a small Llama model for content moderation and nothing else. The Llama Guard 3-8B model card reports that Guard 3 outperforms GPT-4 on safety classification with much lower false positive rates. Llama Guard 4 (12B) is now the current release, folding Guard 3's text and vision variants into a single multimodal model.

Llama Guard classifies by reading the first token's probability as the "unsafe" class probability. An 8B model beats a trillion-parameter one because it isn't trying to be general-purpose. Distilling that teacher into a 1B INT4 version cuts size from 2.8GB to 440MB, at a cost of 3.5 points of English F1 against the 8B; the distillation step itself recovers 1.3 points over standard fine-tuning.

Luna

Luna pushes the approach further. Built on fine-tuned Llama and Mistral models (3B and 8B), it outputs normalized log-probabilities rather than generated text — roughly 150ms evaluations — and stacks LoRA adapters for multi-metric evaluation on one base model.

Luna-2 shows purpose-built beats general-purpose on cost, not on ceiling. Trained for one question — does this response contradict the source? — it matches a frontier judge rather than beating one: the paper benchmarks it against GPT-4.1 with ChainPoll and reports accuracy at par or higher, with the 3B model scoring F1 0.95 on context adherence against GPT-4.1's 0.96, at roughly 80× lower cost. The Splunk Agent Observability documentation puts a Luna-2 evaluator at $0.02 per million tokens — about 125× cheaper than GPT-4o — with F1 0.95 against GPT-4o's 0.94, at 152ms against 3,200ms.

Multi-metric serving with LoRA adapters

The adapter pattern makes several metrics affordable on one GPU. A Llama 3.1 8B base loads once at roughly 16GB; per-metric LoRA adapters of 10-18MB hot-swap in around 5ms. Four adapters — PII, toxicity, compliance, quality — total 55MB against 64GB for four separate models.

PHUDGE

PHUDGE, a fine-tuned Phi-3 model (3.8B parameters), reported state-of-the-art results in 2024 across four evaluation tasks, though its own results lead on two of them.

The key insight: causal modeling is the wrong frame for evaluation. Recast as classification, the model's whole capacity goes to judgment instead of explanation.

Prometheus 2

Prometheus 2 provides an open-weight alternative. Its 7B variant reports Pearson correlations of roughly 0.55-0.67 with GPT-4-1106 evaluations across Vicuna Bench, MT-Bench and FLASK. M-Prometheus (3B, 7B and 14B) followed in 2025 and surpasses Prometheus 2 on English benchmarks. Three techniques transfer: merged training on direct assessment and pairwise ranking produces unified evaluators, swap augmentation reduces position bias, and reference drop improves robustness.

Components of SLM judges

SLM judges are small language models — decoder-only transformers in the 1-8B range — fine-tuned on task-specific data. The base model supplies language understanding, fine-tuning adds judgment, and unlike LLM judges that produce reasoning chains, they often emit only the tokens needed for classification.

Context levels

Context levels determine what gets evaluated. Span-level looks at a single LLM call or retrieval result ("Is this response toxic?") and is fastest and most common in production. Trace-level covers the whole request-response cycle, catching compounding errors in agentic workflows. Session-level spans multiple turns for boundary violations or goal completion, but needs long context, so it runs on sampled traffic.

Output modes

Output modes trade latency for interpretability.

How to build an SLM judge

The build is four stages:

1. Data preparation
2. Model selection
3. Fine-tuning
4. Deployment

We’ll target these metrics: F1 above 95%, P95 latency under 50ms, false positive rate under 2%, 100% coverage, and roughly 50× cost reduction against an LLM judge.

Step 1: Preparing and training data

Your SME-labeled examples from earlier in this series become the training data: ground truth dataset, disagreement analyses and criteria library all translate directly.

Build the test set first — 300-500 manually labeled examples, at least 100 of each class, SME-labeled rather than generated, because it is the ground truth for measuring whether training succeeded. Then target 1,000-10,000 training examples balanced roughly 50/50, even if production is skewed; imbalanced training produces models that default to the majority class.

LLM-generated synthetic examples can expand 1,000 manual labels to 10,000, but only with SME verification: generate candidates, have experts label a sample, keep what aligns with ground truth. Unverified synthetic data encodes the biases you are correcting for.

Hard negatives matter as much as real violations: "I'd recommend diversifying across asset classes" looks like investment advice but is educational content, and without such examples models learn superficial patterns. Record each example's source, labeler, date and criteria version.

Step 2: Model selection

Base model selection matters less than fine-tuning quality: Llama, Phi and Qwen variants are all well-supported, and the differences shrink after task-specific fine-tuning.

Step 3: Fine-tuning

Full fine-tuning updates all weights for the highest accuracy, but costs 8-16 GPU hours for an 8B model and produces 15-30GB checkpoints. LoRA updates only 1-5% of weights with the base frozen: 1-2 GPU hours, 10-50MB adapters, and multiple adapters stacking on one base — the right choice when you have several evaluation tasks.

Configuration: learning rate 1e-5 to 5e-5, batch size 8-32, 3-5 epochs. Validation loss diverging from training loss signals overfitting; stop when validation metrics plateau. Budget 2-5 days for training and 1-2 for validation. A single A100 or L4 fine-tunes an 8B model in hours, spot instances discount 60-80%, and compute costs $50-500.

Step 4: Validation

Hold out 500+ examples from real production traffic, labeled by SMEs. If production F1 differs from test F1 by more than 3-5%, your training data doesn't represent production. For reference, a fine-tuned evaluator lands within five F1 points of the judge it learned from, on 300-500 labeled test samples.

Check calibration too: predictions at 80% confidence should be right 80% of the time, because high confidence on wrong predictions is worse than none. Then run shadow mode — route all traffic through the new SLM while the existing system stays authoritative, and investigate every disagreement. It reveals edge cases no test set captures.

How to serve SLM judges at scale

Once you’ve built the SLM judge, it’s time to serve them. Here are the notes to consider:

Deployment options

There are three deployment, in descending order of control:

• Self-hosted inference with vLLM, TGI, or Triton gives the most control and the lowest per-inference cost at high volume.
• Managed inference through Modal, Replicate or Baseten abstracts the infrastructure away.
• On-premises deployment handles air-gapped environments or strict data residency.

Optimization techniques

INT8 quantization drops precision from FP16 to 8-bit integers and roughly halves memory, and SmoothQuant reports up to about a 1.5× speedup for 8-bit weights and activations. Batching groups 8-32 requests, tripling throughput for async monitoring. Request-aware load balancing matters more than teams expect: naive round-robin wastes 40-60% of GPU capacity when inference time varies with input size.

Multi-metric serving through adapters is what changes the cost structure: a separate model per metric quadruples infrastructure cost, while one resident base model plus hot-swapped LoRA adapters lets a single GPU serve all of them.

Integration patterns

Most production systems combine these three patterns, capturing immediate intervention, operational monitoring and strategic insight:

• Synchronous evaluation blocks until complete — use it for guardrails that must stop harmful content reaching a user.
• Asynchronous evaluation runs after the response, for monitoring.
• Batch evaluation runs on a schedule, for daily quality reports.

Challenges with SLM judges: Why are SLMs harder than they look?

Training pitfalls

Data quality dominates model quality: verify inter-rater reliability first, because if humans don't agree the model can't learn a coherent pattern. Class imbalance sabotages learning — a model shown 95% "pass" examples learns to always predict "pass," and balancing the set leaves it miscalibrated, so post-training calibration is required. Domain shift stays invisible until deployment: accuracy on historical data doesn't predict accuracy on future traffic.

Serving pitfalls

Training a model is machine learning; serving it at scale is systems engineering — GPU memory, request batching, load balancing, failover — and strong ML teams underestimate it. Decide before launch what happens when it fails: fall back to LLM evaluation, to rule-based heuristics, or let traffic through with monitoring. And automate retraining, because models drift.

Failure modes and mitigations

The pattern: SLMs handle the known distribution efficiently, LLMs catch what falls outside it.

Common failures

When it comes to SLMs, there are four failures that occur often:

• Distribution mismatch, where training came from customer support, but the production environment includes sales, legal, and HR.
• Spurious pattern learning, where every PII example happened to be long, so the model learned "long = PII."
• Confidence miscalibration, which erodes trust.
• Catastrophic forgetting, where retraining improves new cases while regressing working ones (hence why we use regression tests).

From LLM to SLM judges: What does the transition look like?

The timeline below follows best-in-class LLM-to-SLM conversion approach: log all LLM calls, curate task-specific data, cluster by task type, fine-tune, iterate. Agentic interactions generate the training data specialization needs.

Teams that have built one SLM judge build the next in 2-4 weeks; infrastructure and expertise transfer. That hybrid is the default in managed tooling, where every evaluator ships in both forms: an LLM judge, and an SLM you switch to when volume demands it.

Understanding your agents at work is key to trusting AI systems. Learn about Splunk Agent Observability and get hands-on with the Splunk Observability Cloud Free Edition today

FAQs about SLM judges