Measure AI by Cost-Per-Outcome

An agent fleet can produce an invoice down to the penny without showing whether the spend bought useful work. While an invoice shows spend, it may hide whether the work was useful.

We have gotten very, very good at measuring tokens. There are dashboards, per-million rates, line items. And the prices keep falling: per-token costs have fallen more than 90% since 2023. GPT-4 launched in March 2023 at $30 per million input tokens and $60 per million output tokens.

By April 2026, low-cost Google Gemini Flash/Lite models were listed around $0.10 per million input tokens, with output pricing varying by source and model; compared with GPT-4’s roughly $30-per-million input-token pricing in 2023, that input price was about 99.7% lower.

However, a new and uncomfortable question arises: if intelligence is collapsing toward free, why is everyone's AI bill going up?

Cost-Per-Outcome Belongs at the Center

Teams still make buying decisions by comparing models by cost-per-token. It's the number on the invoice, so it feels like the truth. It is not. Cost-per-token tells teams what they paid but tells them nothing about what they got.

The problem shows up when two models answer the same hard problem, and both get it right. OckBench research found that two models reaching the same accuracy can differ by 25× in tokens generated: one answering in roughly 1,600 tokens where another spends 42,000. Same outcome. Twenty-five times the billable output – and at equal token prices, potentially twenty-five times the spend. Therefore, if measured only accuracy, the models look identical. However, if measured for what it costs to achieve that accurate outcome, the difference is the entire game.

Thus, the cost-per-outcome metric is key, as it ties spend to whether the work was actually any good.

Why the Old Number Needs More Context

There are three forces that broke the link between tokens and value, all at once.

First, reasoning models have moved the cost from input to output. OpenAI reasoning tokens, Anthropic extended thinking tokens, and Google Gemini thought tokens are billed as output tokens, at the higher rate, and may not match the visible response.

OpenAI's own documentation says a model may generate anywhere from a few hundred to tens of thousands of reasoning tokens depending on the problem. Run seven benchmarks and OpenAI o1 produced more than 44 million tokens where GPT-4o produced around 5.5 million, roughly 8× for the same suite.

Secondly, smaller prompts are not always cheaper, especially due to caching. Per-request cost can now depend more on cache status than raw prompt length.

Finally, consumption has exploded. Google now processes 3.2 quadrillion tokens a month, up from 9.7 trillion two years ago. Falling prices leads to higher consumption – deeper reasoning loops and multi-agent workflows that multiply tokens per task faster than price falls. A January 2026 paper formalized this phenomenon at this infrastructure layer as the Structural Jevons Paradox.

Therefore, just token counts themselves are not sufficient. It’s important to answer the only question that truly matters: Is the money being spent buying good work?

How Cost-Per-Outcome Works

Effective cost can be defined as tokencost per attempt divided by resolution rate. Pay for ten tries, succeed twice, and the real cost is five times the sticker price. For production decisions, token counts alone do not capture cost efficiency in practical deployments. Expected cost per successful task is the metric. The cost per successful task is the metric.

A model that's pricier per token but right on the first pass can be cheaper per outcome than a "cheap" model that flails. OckBench found that two same-size models reaching similar accuracy can differ over 3× in tokens, and that within a model family the smaller model often burns far more tokens for worse accuracy. On a per-token sheet the cheaper model looks like a bargain. On a per-outcome, that sheet is a tax.

This is why discipline is becoming formal. On June 3, 2026, the Linux Foundation announced its intent to launch a Tokenomics Foundation to build open standards for AI cost management, sitting alongside FinOps. AI tokenomics sits alongside FinOps because tokens are abstract and relating them to a business outcome is very hard.

We Don’t Have the Denominator Yet

Cost is easy; outcome is the open problem.

There is no ground truth for subjective quality. Qualities like helpfulness, tone, and appropriateness depend on a user’s goals and risk tolerance. Additionally, no method gives a definitive answer about whether a response is "good enough." As a result, we proxy–and every proxy has a price-accuracy tradeoff.

Human evaluation correlates perfectly with human judgment by definition, and costs over $1,000 per thousand evaluations and takes days. LLM-as-judge runs cheaper and faster at 0.70 to 0.85 correlation. Cheaper still and the correlation falls off a cliff.

The proxies have their own failure modes. There's a rubber-stamp effect, where humans asked to verify a model's answer are more likely to agree with it even when it's demonstrably wrong. And curated test sets can't guarantee production performance, because real users phrase requests in ways your test set never imagined.

Thus, when someone tells you they've solved cost-per-outcome, be skeptical. The math exists, butthe settled production standard does not exist. The gap is definitional.

Why Getting It Right Is Worth the Fight

The alternative is already visible and inexcusable: spending we can't justify and can't defend. Goldman Sachs found that roughly $700 billion of AI investment in 2025 contributed essentially zero to US GDP growth, even as the buildout itself pushed up inflation. MIT found 95% reporting zero return on AI pilots. Those are not the numbers of a field that knows what its money buys.

The moment outcomes can be measured, teams can act. Evidence-based routing already works: RouteLLM showed 85% cost reduction while retaining 95% of GPT-4 quality by sending only the hard 14% of queries to the strong model. An AI tutoring system held 97.1% of premium quality at 71.6% lower cost by escalating on just 18% of queries. These decisions cannot be made on token count alone. Reading cost and quality side by side is the key.

While cost-per-token is the ubiquitous number, cost-per-outcome is the number needed. The teams that win over the next few years will be the ones that builda specific denominator that reflects what "good enough" means for their work, that is instrumented across their traffic and argues cost against quality with numbers instead of vibes.

That's the open problem and it's the one worth solving.