Stop Measuring Agents in Cost-Per-Token
Cost-per-token is the wrong unit for agents. It rewards models that are cheap-and-wrong and hides the tokens you burn on failed runs. The honest metric is Cost Per Resolution (CPR) — total inference spend divided by tasks actually resolved. Here's the argument, the math, and how to instrument it.
Here is a number that means almost nothing for an agent: cost per million tokens.
It's the headline on every pricing page, the column everyone sorts by, the thing founders quote in board decks. For a chat completion it's roughly fine — one prompt in, one answer out, the token count tracks the work. But the moment you wrap a model in a loop that plans, calls tools, reasons, retries, and sometimes fails, cost-per-token stops describing anything you care about. It measures the fuel. You're paying for the trip.
This is a short argument for retiring it — for agents specifically — and replacing it with a unit that can't lie to you the same way.
The lie hiding in per-token pricing
Per-token cost has one fatal property for agentic systems: it charges you identically for success and for failure.
An agent run that burns 40,000 tokens planning, searching, calling three tools, and producing a correct answer costs the same per token as a run that burns 40,000 tokens flailing, looping, and giving up. The pricing page can't tell them apart. Your finance dashboard can't either. Both show "tokens consumed," and both are blind to the only thing that mattered: did the task get done?
Now stack that across a fleet. Say your agent resolves 55% of tickets. The other 45% still ran. They still called the model, still hit the tools, still generated tokens — and they resolved nothing. Per-token accounting quietly folds that 45% of pure waste into a low-looking "cost per million" and calls it efficient. It isn't efficient. It's 45% of your spend lighting up against a wall.
The worst part is what it does to model selection. Sort your options by cost-per-token and you will systematically pick the model that is cheap and wrong over the one that is expensive and right — even when the expensive one is cheaper per outcome. Per-token pricing has a built-in bias toward exactly the wrong choice.
The metric that doesn't lie: Cost Per Resolution
We coined a name for the unit that does survive contact with agents: Cost Per Resolution (CPR).
Cost Per Resolution is the total inference spend to successfully complete one unit of work, divided by the number of tasks actually resolved — not attempted.
The formula, two ways:
CPR = total inference spend over a period ÷ tasks successfully resolved
CPR = (average cost per attempt) ÷ (resolution rate)
The second form is the one that changes how you think. Divide by the success rate and the failures price themselves in automatically. Watch what happens to the "obvious" cheap choice:
| Cost per attempt | Resolution rate | CPR | |
|---|---|---|---|
| Cheap model | $0.004 | 55% | $0.0073 |
| Strong model | $0.011 | 92% | $0.0120 |
On per-token math the cheap model wins by 2.75×. On CPR it's a far closer call — and a small accuracy swing flips it outright. Push the cheap model to a real-world 45% and it's $0.0089; let the strong model hit 95% and it's $0.0116. The gap collapses, and in plenty of production settings it inverts. CPR is the only number that can see this. Cost-per-token never will.
(The canonical definition, with the full worked example, lives in the inference cost economics guide. This post is the argument for why you should care.)
Why per-token won't die on its own
If CPR is so obviously better, why is every dashboard still per-token? Three reasons, none of them good:
- It's what the providers sell. Per-million-token is the billing unit, so it's the number that shows up by default. Convenient for invoicing is not the same as meaningful for decisions.
- Resolution is harder to measure. Tokens are handed to you; "was this task actually resolved?" requires you to define success and instrument for it. That's real work — and it's exactly the work that distinguishes teams who know their unit economics from teams who are about to be surprised by them.
- It flatters everyone. A low cost-per-token looks great in a deck. CPR is less forgiving — it surfaces the failed runs you were quietly paying for. Nobody volunteers a metric that makes their numbers worse, even when it makes their decisions better.
The throughline: per-token persists because it's easy and flattering, not because it's right.
How to instrument CPR this week
You don't need a platform. You need one honest definition and one extra field in your logs.
- Define "resolved" before you measure anything. This is the whole game. A resolved task is one that met your success bar — the ticket closed without escalation, the PR merged, the answer passed your eval, the user didn't immediately retry. Pick the bar your business cares about, write it down, and make it binary per run.
- Log cost and outcome on the same span. One record per task with
total_cost_usdandtask_success(plusmodel,tool_calls,retriesfor slicing later). The OpenTelemetry GenAI conventions standardize the field names. - Compute it as
total_cost_usd ÷ count(task_success = true)over a rolling window, segmented by task type. That's CPR. Track it the way you track any unit-cost KPI. - Watch resolution rate and CPR together. A CPR that drops because resolution climbed is real progress. A CPR that drops because you switched to a cheaper-but-flakier model is a mirage — the savings are sitting in the failed runs you stopped counting.
The objections, answered
"Per-token is fine for chat." Agreed — for a single-shot completion where the work and the token count move together, per-token is a reasonable proxy. CPR is an agent metric. The argument is scoped to systems that loop, branch, and can fail.
"Resolution rate is fuzzy." Sometimes. But fuzzy-and-relevant beats precise-and-irrelevant. A rough binary success signal you argue about is worth more than a perfectly precise token count that doesn't predict anything. Start coarse; sharpen the definition as you learn what "resolved" really means for your product.
"We already track success rate separately." Then you're one division away from CPR — and you're leaving the most decision-relevant number uncomputed. Success rate tells you if it works; cost-per-token tells you what the fuel costs; CPR is the only one that tells you whether it's worth running. Don't keep them in separate dashboards where nobody multiplies them.
CPR's companion: Time To First Action
CPR answers "is it worth running?" The other agent question is "does it feel alive?" — and per-token can't touch that either. Its latency companion is TTFA — Time To First Action: the wall-clock from a user's request to the agent's first observable move. CPR is the economic metric; TTFA is the experiential one. Between them they describe an agent better than any per-token figure ever could.
The bottom line
Cost-per-token survived the chat era because the work and the tokens lined up. Agents broke that link. A run that fails costs real money and resolves nothing, and any metric that prices it the same as a success is lying to you in the one place — model selection — where the lie is most expensive.
Measure what you actually buy. You're not buying tokens; you're buying resolved tasks. Price them that way.
Stop sorting by cost-per-token. Start tracking Cost Per Resolution.