How to Reduce Agent Cost by Model Routing

The model you choose for an agent sets a price on every decision it makes. Put a frontier model behind the entire workflow, and even the routine steps inherit frontier-model economics.

In the first post in this series, What is Agent Tokenomics?, we sorted the cost of an agent call into five token buckets. Model selection touches all five, which makes it a natural place to begin cutting cost.

Most teams start with the strongest model they can afford and use it everywhere. That is a sensible way to get an agent working. It becomes expensive when the same choice quietly survives into production, long after the team has enough data to see which parts of the workflow are easy.

Different Steps Need Different Levels of Intelligence

Consider a customer-service agent handling a refund. It must:

• Interpret the request
• Retrieve recent orders
• Check the policy
• Call the right tool
• Validate the result
• Write a response

One of those steps may involve a genuinely ambiguous policy decision. Several others are narrow and predictable.

Sending all of them to the same frontier model is simple, but simplicity carries a recurring premium. The model that is valuable for the ambiguous decision is often unnecessary for extracting an order number or checking whether a required field is present.

This gives teams two ways to approach model selection. Passive model selection means choosing one model for an entire workload. It is easy to build, easy to evaluate, and often the right way to ship a first version. Active model selection, or routing, chooses a model for each step based on the difficulty and consequence of that step.

A common pattern is to keep a frontier model in charge of planning and high-consequence decisions, then send well-scoped execution work to smaller models. The expensive model sees short prompts and makes fewer calls. Cheaper models process most of the token volume. You do not need to design this split in advance. Start with one model, trace the workflow, and add routing as the evidence suggests.

A Mixed-Model Workflow Changes the Economics

Routing is attractive because model prices span such a wide range. Input tokens can range from about $0.05 per million for the least expensive models to roughly $3 for a frontier model. That is more than a 50-fold spread for just reading that can easily explode on research intensive tasks.

The research quality difference on a routine task can be much smaller than the price difference. A few points of accuracy may be acceptable on a low-stakes step with an inexpensive recovery path. The same drop can be disastrous during a planning phase, where one weak decision sends the rest of the session down the wrong path. Good routing accounts for both task difficulty and the cost of being wrong.

The three Claude tiers make the idea concrete. At current list pricing, the rate rises at each step from Haiku to Sonnet to Opus. Running the whole workflow on Opus charges the highest rate for every token, regardless of what those tokens are doing.

A planning call may use only a few hundred tokens, while an extraction step reads a document that is tens of thousands of tokens long. If Haiku handles most of that document work, Sonnet takes the middle tier, and Opus is reserved for the few strategy decisions that warrant it, the blended rate can fall sharply.

Your code already knows whether it is planning, extracting, validating, or drafting. That step label is enough to make an initial routing decision.

Proven Ways To Route Model Requests

There are three practical ways to route model calls. They build on one another, so the right starting point is usually the least complicated option that captures meaningful savings.

1. Rules Route Known Work.

A rules-based router like Routerly uses signals the application already has: task type, prompt length, health, or budget. Planning goes to a frontier model; extraction and validation go to a smaller one. Rules are transparent, quick to change, and often capture a large share of the available savings.

2. Cascades Escalate Uncertain Answers.

A cascade sends a step to the cheapest suitable model first, then promotes the request when a scorer rejects the response. FrugalGPT, a 2023 Stanford paper, matched GPT-4 quality at up to 98% lower cost on its benchmarks using this general approach. The scorer is the hinge: if it approves weak answers or rejects good ones, quality or cost will suffer.

3. Learned Routers Predict the Right Tier.

A learned router uses a small classifier to choose a model before the request runs. RouteLLM, which was trained on human preference data, reported savings of up to 3.66x while retaining 95% of GPT-4's quality on MT-Bench. Its advantage narrowed on harder benchmarks, a useful reminder that routing results depend on the workload. This approach starts to make sense when call volume is high enough to justify training and maintaining components.

Whichever method you choose, keep the policy in one place. A gateway between the agent and its model providers can apply routing rules, record spend by model, and change providers without forcing every application to implement the same logic. The FinOps Foundation describes this as applying FinOps discipline to AI: managing the variable cost of intelligence with the same care used for other cloud resources.

The Handoff Determines How Much Quality Survives

Choosing a cheaper model is only part of the design. The way you package its work can matter even more.

Minions, a 2025 Stanford study, paired a small on-device model with a frontier cloud model for long-document tasks. The researchers compared two ways of dividing the work. In the first, the models exchanged messages over several turns. Remote cost fell by 30x, but the system retained only 87% of the frontier model's quality. The small model struggled to follow multi-step instructions across a long context.

The second approach gave the frontier model a different role: break the problem into focused subtasks, then let the local model run them in parallel. Quality rose to 97.9% of the frontier baseline, with a 5.7x cost saving. Changing only the handoff produced a much stronger quality-cost tradeoff.

That result captures a broader design principle. Small models tend to perform well when the task has a clear boundary, a compact input, and an answer that can be checked. Their reliability falls when the work requires them to preserve an ambiguous plan across a long chain of steps. Routing therefore works best alongside decomposition. Give the cheaper model a job it can finish cleanly, then validate the result before the workflow continues.

Your Workload Has the Final Say

Pricing tables and public benchmarks are useful for building a shortlist. They cannot show how a model will behave on your prompts, tools, data, and failure modes. Two practices close that gap. Tracing records the cost, latency, and outcome of every step, so you can see which ones are worth touching in the first place. Evaluation scores whether a model's output meets the bar, so the decision to switch rests on measured quality rather than a quick read of a few examples.

With both in place, you can test the choice directly. Take a step that looks over-provisioned and run a representative sample through both the current model and a cheaper candidate. Score both with the same eval against the same acceptance criteria, then examine the average result alongside the individual failures. A model that misses a small set of easy-to-recover cases may be a good trade. A model that fails less often but creates rare, expensive incidents may be a poor one, and only an eval over enough real traffic will separate the two.

This is where price per token gives way to cost per accepted task. A model can cost 80% less per call and still lose money if it triples failure-rate or sends more work to human reviewers. The full measure includes model spend, tool and runtime costs, retries, and human review.

A cheaper model is genuinely cheaper only when those outcome measures improve.

Where To Start This Week

Open the workflow with the highest token spend and write down each step. Choose three operational steps that currently run on a frontier model, then replay a representative sample on a cheaper model. Compare acceptance rate, retry rate, review time, and total cost. Move the step only when cost per accepted task falls without introducing a failure mode you cannot tolerate.

The decision is not permanent. Models drift, providers release new ones, and your own traffic shifts over time. Keep the eval running on a sample of live calls and keep the traces live on cost and quality for each step. When a routed step starts to slip, you want it to surface on a dashboard rather than in a customer complaint.

Check Your Own Tokenomics for Free

If you code with Claude Code or Codex, you can watch these numbers live. Token Meter, an open-source tool from Galileo Agent Labs (now a part of Cisco), follows your local agent logs and streams the cost of a run to a dashboard on your own machine. It runs entirely local, no API keys, no telemetry, so it is a low-risk way to see where your tokens go before you change anything. Point it at your next long run and watch the burn profile.

Token Meter's menu-bar widget is a glanceable companion to the full dashboard: it lives in the macOS menu bar and surfaces the state of your active agent session at a glance.