Up to 98% cheaper changes the economics
Each evaluation predicts a single token, so verdicts cost up to 98% less than a frontier LLM judge and return in about a third of a second. Grade and guard every interaction, not a sample.
Eval-G1
Eval-G1 evaluation models
Purpose-built small language models, tuned to your rules, that evaluate and guardrail 100% of agent traffic in about a third of a second.
Too expensive to grade everything
A frontier LLM judge writes out its reasoning before it gives a verdict, often hundreds of tokens per evaluation. At production volume that is too expensive to run on everything, so teams review a sample and hope the rest look the same. The ones you miss are the ones that hurt: the leaked account number, the toxic reply, the broken policy.
Too slow to stop anything
A frontier LLM judge takes about 2 to 5 seconds to return a verdict. By then the answer has reached the customer, so the verdict can only describe what happened. Eval-G1 returns a verdict in about a third of a second, up to 14x faster, so a runtime guardrail can stop a policy violation before it lands.
Deterministic, single-token scoring
Eval-G1 returns a verdict as a single token with a calibrated probability, so scores are fast and repeatable. Run the same input twice and you get the same answer, which is what makes a model trustworthy as an evaluator.
One base model, many evaluators
Every Eval-G1 evaluator is a small adapter on one shared base model, so adding an evaluator doesn’t add a model, a server, or an MLOps team. Splunk runs each one in production for you, ready to score and guard in real time.
features
Tune evaluators to your rules
Find the failures that matter
Signals in Splunk Agent Observability flags failure patterns across your agent traffic and traces each one to the conversations behind it, such as an assistant that keeps misapplying a policy, so you know which evaluator to build next.
Let your experts define correct
Your experts mark real conversations pass or fail and write the rule down as a short rubric. Their labels set the standard every verdict follows, which makes this the step that matters most.
Build training data from a few labels
The evaluation tuner grows a few dozen expert labels into thousands of training examples, covering rephrasing, edge cases, and near misses, so the evaluator learns the hard calls too.
Tune a custom evaluator model, no code
The evaluation tuner tunes a custom evaluator model on those examples while your team gets on with its day. No data science team, labeling pipeline, or model training expertise required.
Validate against your LLM judge
The evaluation tuner compares the new evaluator with your LLM judge on conversations neither has seen. Review each disagreement, then register the evaluator when it meets your standard.
Use it as an eval or a guardrail
Your custom evaluator appears in Splunk Agent Observability alongside your other evaluators. Score production traffic with it, or run it as a runtime guardrail, with each verdict in about a third of a second.
Resources
Explore more from Splunk
Eval-G1 Evaluation Models FAQs
Eval-G1 is a purpose-built small language model for evaluation and the base for every custom evaluator model you tune. It grades and guards agent interactions quickly and cheaply, which is what makes it affordable to watch 100% of traffic instead of a sample.
A frontier judge is expensive and slow, so teams sample 1 to 5% of traffic and can only review actions after they run. Eval-G1 is built for one job, judging agent interactions, so it is cheap enough to grade everything and fast enough to guard in real time.
Eval-G1 runs evaluations and guardrails at up to 98% lower cost than a frontier LLM judge, which changes the economics from rationing to running on every interaction.
Yes. Eval-G1 returns a verdict as a single token, so scores are deterministic: run the same input twice and you get the same answer, which is what makes a model trustworthy as a judge.
Yes. In evaluation tuner, your experts label real conversations and write the rule down. evaluation tuner turns those labels into a custom evaluator model, reaching up to 0.95 F1, with no data science team or model-training expertise.
Related products
Splunk Enterprise Security
Unified threat detection, investigation, and response for the agentic SOC.
Splunk IT Service Intelligence
Predict and prevent IT issues with AI-driven service monitoring.
