7 LLM Metrics to Enhance AI Reliability

A model can be fast, cheap, and confidently wrong. Because a prompt can have dozens of valid answers, LLMs need evaluation across multiple dimensions.

What are LLM performance metrics?

Performance metrics for large language models (LLMs) quantify model capabilities, weaknesses, and improvement across dimensions.

Traditional machine learning (ML) uses deterministic metrics like accuracy and precision, assuming one correct answer exists. LLMs create a many-to-many relationship between prompts and acceptable responses — ask "summarize this article," and dozens of summaries can be valid. Evaluation must assess factual accuracy, relevance, coherence, safety, creativity, and efficiency together.

Limitations of traditional metrics for generative AI

String-matching metrics, including bilingual evaluation understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE), miss semantic equivalence. A factually correct answer using different vocabulary from the reference can receive a low score, and studies of hallucination detection have found that overlap metrics such as ROUGE can correlate poorly with factuality judgments.

LLM-as-a-judge is common for open-ended tasks. At the International Conference on Learning Representations (ICLR), the JudgeBench paper at ICLR 2025 showed the strongest judge model reaching only 64% accuracy, and the same survey puts untuned LLM judges at 66–68% agreement with human experts. Judges need tuning and calibration against human labels before use.

Top LLM metrics for improving reliability

1. Latency

Latency is the delay between submitting a prompt and receiving the complete response. Interactive users expect milliseconds, not seconds.

For streamed generation, an end-to-end number hides where the time goes. NVIDIA's benchmarking guide splits latency into time to first token (TTFT), the delay before the first token appears, and inter-token latency. This is computed as:

(end-to-end latency − TTFT) / (total output tokens − 1)

Percentile conventions vary by platform, so fix yours before comparing numbers. For chat, 250ms average TTFT as a target. Reasoning models complicate this because TTFT includes hidden thinking before the first streamed token.OpenAI's prompt caching reduces latency by up to 80% on cache hits for prompts of 1,024 tokens or more. Splunk Agent Observability logs latency and tokens on LLM spans and charts estimated cost, latency, and input, output, and total tokens.

2. Throughput

Throughput measures capacity in tokens per second, requests per minute, or queries per second across concurrent requests.

Scheduling is the biggest software lever. Continuous batching and paged key-value (KV) memory, introduced in the vLLM PagedAttention paper, delivered two to four times the throughput at the same latency over earlier serving systems, and vLLM's V1 engine sustains 2,200 tokens/s per H200 GPU with chunked prefill on by default. Published peak numbers assume tuned batching, so measure throughput at your concurrency levels.

3. Perplexity

Perplexity measures a model's "surprise" when encountering test data. Hugging Face defines it as the exponentiated average negative log-likelihood of a sequence, equivalent to exponentiating the cross-entropy between the data and the model's predictions. Lower perplexity indicates better prediction. Perplexity is only comparable across models with identical vocabularies, as Jurafsky and Martin note.

In controlled pretraining regimes perplexity tracks downstream quality well; Gadre et al. fit a power law across 104 models that predicted average downstream error to within one percentage point. It breaks in other settings. Hu et al. found no correlation between perplexity and long-text understanding. Treat perplexity as a health check, not a standalone quality metric.

Splunk Agent Observability's Prompt Perplexity metric applies the same idea to inputs, using the model's log probabilities to measure how familiar a prompt is. Tracked over time, it flags text the model handles poorly.

4. Cross-entropy

Cross-entropy measures the difference between predicted token probability distributions and actual data distributions. It is the loss function used in training and evals, and lower values indicate better alignment between predictions and targets.

Perplexity is cross-entropy exponentiated, so the two carry the same information in different units. Token-level analysis shows which vocabulary items or constructs the model gets wrong.

Cross-entropy shares perplexity's limitations as a proxy for task-specific performance, and sometimes moves in the opposite direction. Isik et al. observed translation scores behaving non-monotonically while downstream cross-entropy kept improving smoothly, and warned that "using cross-entropy as a proxy for task-related metrics like BLEU, ROUGE, or COMET scores may lead to critical misjudgments in practice." Pair it with task-level accuracy metrics rather than reading it alone.

5. Token usage

Token usage counts the tokens processed during inference, driving operational cost and context window efficiency. At list prices, GPT-5 costs $1.25 per million input tokens and $10.00 per million output tokens, so output tokens are priced eight times higher than input.

Track input and output tokens separately. Input efficiency means achieving the result with minimal prompt length; output efficiency means responding concisely while maintaining quality. Reasoning models add a third bucket: OpenAI's reasoning.effort setting and Anthropic's budget_tokens cap how many hidden thinking tokens you pay for.

Windows have grown, with GPT-5 accepting 400,000 tokens and Claude 4.6 and later models accepting 1,000,000, but every context token adds cost and latency. Monitoring utilization surfaces opportunities for prompt compression, truncation, or a smaller model.

Prompt caching cuts repeated input cost on OpenAI by billing cached input tokens at a discounted rate, and OpenAI and Anthropic price asynchronous batch work at 50% of standard rates.

Splunk Agent Observability logs input, output, and total token counts on every LLM span, and admins can set per-model prices so cost appears next to latency in trend charts.

6. Resource utilization

Resource utilization covers GPU/tensor processing unit (TPU) computation, memory, CPU usage, and storage, influencing cost, deployment flexibility, and environmental impact.

GPU memory limits deployment options. PyTorch torchao reports Llama-3-8B running 1.89x faster with 58% less memory at four-bit integer (INT4) precision. In training, activation checkpointing trades roughly 20% slower steps for lower activation memory.

On the serving side, the PagedAttention work showed prior systems using only 20.4%–38.2% of allocated KV-cache memory, versus under 4% waste with paging. NVIDIA multi-instance GPU (MIG) partitions a GPU into up to seven instances with hardware-isolated memory and fault domains; time-slicing provides no such isolation.

Google's production measurement of Gemini Apps put the median text prompt at 0.24Wh, with only 58% going to the AI accelerator and the rest to host CPU, idle machines, and data center overhead.

7. Error rates

Error rates measure request failures, timeouts, malformed outputs, and disruptions across the request lifecycle. Categorize failures into types:

• Infrastructure errors (hardware/network failures)
• System errors (software crashes, memory exceptions)
• Model errors (hallucinations, reasoning failures)
• Integration errors (API mismatches, serialization issues)

An analysis of 156 high-severity LLM inference incidents attributed roughly 60% to inference-engine failures, with timeouts making up about 40% of those, and only about 74% were caught by automated detection. Model errors are harder to count: the RIKER benchmark found every tested model exceeding a 10% fabrication rate at 200,000-token context. ReliabilityBench found agents scoring 96.9% pass@1 under clean conditions dropped to 88.1% with modest fault injection, so test against injected faults.

Set error budgets and alert automatically when you approach them. Google Cloud's AI/ML reliability guidance gives service-level objective (SLO) targets of 99.9% successful API calls and a harmful-output rate below 0.1%, with burn-rate alerts on the budget. Log errors with context, model configuration, and environment details, and audit detection itself: a study of Microsoft's GenAI cloud incidents found 38.3% of incidents were reported by humans rather than monitors.

For high-reliability applications, the AWS Well-Architected Reliability Pillar recommends the circuit breaker pattern to avoid remote calls that are timing out, and the Google site reliability engineering (SRE) Book describes load shedding and degraded modes for cascading failures.

Signals in Agent Observability analyzes production traces to detect failure patterns, classifying each into Errors, Warnings, Suggestions, or Enhancements and linking to the span where the problem occurred. A single click turns a signal into an LLM-judge eval, so the same failure pattern is evaluated. Tool-error and task-completion metrics for agents sit on top of these raw error rates.

Improve your LLM performance with Splunk

An LLM performance eval program combines the operational metrics with generation-quality and safety measures. Splunk Agent Observability covers both sides for teams building LLM applications and agents:

• Operational metrics on every trace: Latency and token counts logged per span, cost and token trends on the Trends page, and auto-capture through OpenTelemetry and OpenInference.
• Quality metrics out of the box: 31 across eight categories, including nine agentic metrics such as Tool Selection Quality and Action Completion.
• Luna evaluation models: Small language models purpose-built for evaluation that score 10–20 metrics simultaneously in under 200ms at up to 96% lower cost than frontier LLM-as-judge.
• Signals andAutotune judge prompts: Automatic failure-pattern detection with severity ranking and one-click eval generation, plus Autotune, which adapts a judge prompt from your corrections and is documented to raise metric accuracy by 20–30%.

Splunk's mission has always been to provide the visibility that keeps digital systems secure and reliable. Agents are part of those systems, and they fail in ways a dashboard of request counts won't catch. Instrumenting, evaluating, and governing them is what it takes to be the intelligence layer for trusted operations.

Read The Agentic Shift: Redefining Observability for the AI Era to see what agent-era observability requires.

FAQs about LLM metrics