Prompt20
NewsDataBlog
All posts
finopstoken-spendcost-managementagentsgovernanceobservabilityhow-toevergreen

AI FinOps: How to Manage and Govern Token Spend

A practical playbook for running FinOps on AI: where token costs come from, why agent workloads blow past budgets, and how to instrument, allocate, cap, and govern spend before the bill becomes the story.

By Prompt20 Editorial · 22 min read

Every AI feature you ship has a meter running behind it. Each call sends tokens in and gets tokens out, and each token has a price. For a single chatbot reply the cost is a fraction of a cent, small enough to ignore. Multiply it by an agent that loops twenty times, re-reads a growing context on every step, and runs across ten thousand users a day, and the fraction of a cent becomes a line item your finance team asks about by name. AI FinOps is the practice of making that spend visible, attributable, and controllable before it becomes the thing everyone is talking about.

This guide is the buyer-side and operator-side counterpart to model unit economics. It assumes you are already paying for AI and now need to run it like a budgeted system: know where the money goes, tie it to teams and features, cut the waste, and put guardrails around the parts that can run away. The mechanics are specific to tokens, but the discipline is the same one cloud teams learned a decade ago when EC2 bills started arriving with surprises.

Key takeaways

  • Tokens are the unit of spend. Request counts do not track cost: two calls with the same request count can differ 100x depending on context size and output length.
  • Agents are cost multipliers. A single user action can trigger dozens of model calls, each carrying the full growing context, so agent spend scales with the square of a naive design.
  • You cannot govern what you cannot attribute. The first real FinOps step is tagging every call with a team, feature, and user so the bill decomposes into something actionable.
  • The biggest levers are context size, model choice, and caching. Trimming context, routing easy calls to cheap models, and caching stable prefixes cut most of the waste without touching quality.
  • Budget with hard caps and rate limits. Per-key and per-feature limits turn a runaway loop into a handled error instead of a five-figure surprise.
  • Measure cost per resolved task rather than cost per token. A cheaper token that fails and retries three times ends up more expensive than a pricey token that works once.

Table of contents

What AI FinOps actually is

FinOps started as a cloud discipline: a shared practice between engineering, finance, and product for managing variable spend on infrastructure you rent by the second. The core loop is inform, optimize, operate. Inform means everyone can see what things cost and who is spending. Optimize means you find and remove waste. Operate means you set budgets and policies and hold to them.

AI FinOps applies that loop to model consumption. The variable is now tokens processed by a model provider or an inference cluster you run, in place of instance-hours or gigabytes of egress. The shape of the problem rhymes with cloud FinOps, so the muscle memory transfers. The details differ enough that a cloud FinOps dashboard will not tell you why your support agent cost tripled last week. That answer lives in token accounting, and token accounting has its own rules.

The reason this became urgent in 2026 is agents. A retrieval chatbot has a predictable cost per message. An agent that plans, calls tools, reflects, and retries has a cost that depends on how hard the task turned out to be, which you do not know in advance. Spend went from a smooth, forecastable curve to a spiky, workload-dependent one, and the old habit of checking the monthly invoice stopped being good enough.

Where token costs come from

To manage the bill you have to read it, and the bill has more than one number. A model call is priced across several dimensions, and most teams only think about one of them.

Input tokens. Everything you send: the system prompt, retrieved documents, conversation history, tool definitions, and the user's message. Input is usually the cheaper per-token rate, but in a long-context or agent workload it is often the larger share of the bill because the context is big and gets resent on every step.

Output tokens. Everything the model generates. Output is typically priced several times higher than input, often four to five times. A verbose model, or one asked to "think step by step" at length, spends real money in the output column. Reasoning models that generate long internal chains before answering can produce many hidden output tokens you still pay for.

Cached input. Most providers now offer prompt caching: if a large prefix of your input is identical to a recent call, they charge a fraction of the normal input rate to reuse it. Cache reads can be a tenth or less of the base input price. This is the single most underused cost lever, because it requires you to structure prompts so the stable part comes first.

Context length tier. Some models charge more per token once the context crosses a threshold, for example above 200K tokens. A workload that quietly drifts into the long-context tier can see its unit price step up without anyone changing the model.

Batch and off-peak rates. For work that does not need an immediate answer, batch APIs often run at half price. Nightly summarization, backfills, and evals are prime candidates that teams routinely pay full price for out of habit.

The practical consequence is that two features with identical call counts can have wildly different costs. A feature that sends a 50-token prompt and gets a 100-token answer is nearly free. A feature that stuffs 40K tokens of context into every call and lets the model ramble for 2K tokens of output, with no caching, costs a hundred times more for the same number of requests. Request counts lie. Token accounting tells the truth.

Why agents blow the budget

The defining cost pattern of modern AI is the agent loop, and it is worth understanding exactly why it is expensive, because the fix follows directly from the cause.

An agent works by calling the model repeatedly. It plans, calls a tool, reads the result, decides what to do next, and repeats until it thinks it is done. Each of those steps is a full model call. A task that takes fifteen steps means fifteen model calls. That alone is a 15x multiplier over a single-shot feature.

The worse problem is context growth. On each step, the agent typically resends the entire history so far: the original instructions, every prior tool call, and every prior result. Step one sends a small context. Step ten sends everything from steps one through nine. If each step adds roughly the same amount of context, the total tokens processed across the whole run grow with the square of the number of steps. A run that is twice as long can cost four times as much. This is why one hard user request, the kind that sends the agent down a long path, can cost more than a hundred easy ones.

Tool definitions add a fixed tax. Every call carries the full schema of every tool the agent can use, resent on each step. An agent with thirty tools defined pays for those thirty schemas fifteen times over in a fifteen-step run, whether or not it uses them.

Retries and reflection compound it further. An agent that second-guesses itself, or a harness that retries failed steps, multiplies the call count again. None of this is visible in a request-per-user metric. It shows up only when you measure tokens per completed task, and when you do, the distribution is brutally long-tailed: a small fraction of hard tasks consume most of the budget.

The takeaway is not that agents are bad. It is that agent spend is a design output. How you manage context, how many tools you expose, how you cap loop length, and which model runs each step all move the bill by large factors. FinOps for agents is mostly the discipline of controlling those factors on purpose.

Step one: make spend visible

You cannot optimize a number you cannot see. The first concrete move in AI FinOps is instrumentation, and it comes before any clever cost cutting, because without it you are guessing.

At minimum, every model call your system makes should be logged with: the model used, input tokens, output tokens, cached tokens, cost, latency, and a set of tags that say who and what the call was for. The provider's dashboard gives you a total and maybe a per-key breakdown. That is not enough to act on, because it does not tell you which feature, team, or user drove the spend.

The fix is to route every call through a thin layer you control, a gateway or a wrapper, that attaches metadata and records the token counts the provider returns. Providers return usage figures on every response. Capture them. Once you have a per-call record with tags, you can answer the questions that actually matter: which feature is most expensive, which is growing fastest, what a single user session costs, and how cost per task moved after a change.

Three views turn raw logs into decisions:

  • Cost by feature over time. Shows you where the money is and which features are trending up. This is where you point optimization effort.
  • Cost per unit of work. Cost per conversation, per resolved ticket, per document processed. This normalizes for growth so you can tell a real efficiency change from just more usage.
  • The full spend distribution, including the tail. The mean cost per task hides the long tail. Plot the p95 and p99. The expensive tail is usually where a cap or a routing change pays off most.

If you build nothing else, build the per-call log with tags. Everything downstream depends on it.

Attribution and chargeback

Visibility becomes governance when spend is attributed to the people who can change it. In a small team one shared API key is fine. Past that, a single key is a black hole: you know the total and nothing else.

The pattern that scales is a key or a tag per meaningful unit: per team, per product, per environment, and ideally a propagated identifier for per-feature and per-user attribution. With that in place, the monthly number decomposes. Finance can see that the support product spends 60% of the AI budget and the internal tools team spends 5%. Product can see that one feature is 80% of a product's cost. Engineering can see that staging is somehow spending as much as production, which usually means a test loop left running.

Chargeback, or at least showback, changes behavior. When a team sees its own AI spend attributed to it, the incentive to trim waste arrives on its own. The point is not to bill teams internally for its own sake. The point is that attributed spend creates ownership, and owned spend gets managed. Unattributed spend is nobody's problem until it is everybody's problem on the invoice.

The levers that cut spend

Once you can see and attribute spend, cutting it is a matter of pulling a few high-leverage levers. In rough order of payoff:

Trim the context. The cheapest token is the one you never send. Most prompts carry passengers: stale conversation history, over-retrieved documents, verbose tool schemas, boilerplate instructions. Retrieve fewer and better passages. Summarize old turns instead of resending them verbatim. Prune tool definitions to the ones a given step could plausibly need. In agent workloads this is usually the single biggest win because context is resent on every step, so every token you cut is paid back many times.

Route to the right model. Not every call needs your most capable model. Classification, extraction, routing, and simple formatting run fine on a small cheap model at a fraction of the cost. Reserve the frontier model for the calls that genuinely need it. A router that sends easy calls to a cheap model and hard calls to an expensive one can cut spend by half or more with no visible quality loss, because most calls in most systems are easy.

Cache stable prefixes. Structure prompts so the unchanging part, the system prompt, the tool definitions, the fixed context, comes first and the variable part comes last. Then enable prompt caching. Repeated calls that share that prefix pay the cache rate on it, often a tenth of the base. For agents that resend a large stable instruction block on every step, this is close to free money.

Cap output length. Verbose output is expensive output. Set max output tokens deliberately. Ask for structured, terse responses when you do not need prose. For reasoning models, use the effort or budget controls to stop them from over-thinking simple tasks.

Batch what can wait. Move evals, backfills, summarization, and any non-interactive workload to the batch API and off-peak windows. Half price for work nobody is waiting on is an easy yes.

Bound the agent loop. Set a maximum step count and a maximum token budget per task. A task that has not finished in fifteen steps is usually stuck, and letting it run to fifty is pure waste. Fail it, log it, and handle it rather than paying for the flailing.

Each lever is independent, so they stack. A team that trims context, routes calls, and caches prefixes routinely lands a two-thirds reduction in spend for the same output, which is the difference between an AI feature that pencils out and one that does not.

Governance: budgets, caps, and guardrails

Optimization lowers the average. Governance protects you from the tail: the runaway loop, the abusive user, the deploy that accidentally 10x's a prompt. These are the events that produce the horror-story invoices, and they are prevented by policy, not by good intentions.

Hard caps per key and per feature. Set a spending limit that, when hit, stops the calls. A cap that turns a runaway into a handled error is worth more than any dashboard, because dashboards tell you about the disaster after it happened and a cap prevents it.

Rate limits per user. Bound how much any single user or session can consume in a window. This stops both abuse and the honest bug where a client retries in a tight loop.

Anomaly alerts. Alert on spend that deviates from the baseline: a feature whose cost per task jumps, a sudden volume spike, an environment that starts spending when it should be idle. The goal is to hear about a runaway in minutes, not on the monthly bill.

A budget with an owner. Every product or team with AI spend should have a monthly budget and a named owner who watches it. Budgets without owners are decoration.

Change review for prompt and model changes. A one-line prompt edit that adds 10K tokens of context, or a model swap to a pricier tier, can move the bill more than a quarter of feature work. Treat prompt and model changes as cost-affecting changes, and look at the projected spend impact before they ship.

Governance is where AI FinOps stops being a reporting exercise and becomes a control system. The reports tell you where you are. The caps and limits keep you from driving off the cliff while you read them.

The metric that matters: cost per resolution

The most important shift in AI FinOps is choosing the right denominator. Cost per token is the price you pay. It is not the thing you should optimize, because it rewards the wrong behavior.

Consider a cheap model that costs a fifth as much per token but fails a quarter of the time, so the system retries, escalates, or hands off to a more expensive model. The per-token price looks great. The cost to actually resolve the task, counting the retries and the escalation, is worse than using the capable model once. Optimizing cost per token pushed you toward the more expensive outcome.

The metric to run the practice on is cost per resolved task: cost per answered question, per closed ticket, per correctly processed document, whatever a completed unit of work is in your domain. It counts everything the task consumed, including the failures and retries, against the outcome you actually wanted. It aligns the cost you are cutting with the value you are delivering.

This reframes every lever above. A cheaper model is a win only if cost per resolution drops, which means quality held. A shorter context is a win only if the task still succeeds. Measuring resolution keeps optimization honest, because it makes a quality regression show up as a cost increase, which is exactly where a FinOps practice will notice it.

Build vs. buy: the cost-tooling landscape

You can run AI FinOps with tooling you build or tooling you buy, and most teams end up with a mix.

The build path starts with the per-call log described above, a dashboard on top of it, and caps enforced in your own gateway. This is not a large project, and it has the advantage that the metadata and the controls live where your code already is. For a single team with one main workload, building is often the right call because the whole thing is a few hundred lines and a table.

The buy path is a growing category of AI cost and observability platforms that sit between your app and the providers, capture usage automatically, and give you dashboards, per-team attribution, alerts, and sometimes routing and caching built in. They earn their keep when you have many teams, many providers, and no appetite to maintain the plumbing. The trade is the usual one: faster to stand up, less control over the internals, and a dependency in your critical path if they enforce limits inline.

A reasonable default is to build the per-call log and caps yourself, because they are small and belong close to your code, and buy the cross-team reporting and attribution layer once more than a couple of teams are spending real money. Whichever way you go, the requirements are the same: capture usage on every call, attribute it, show it per unit of work, and enforce hard limits. The tool is a means to those four things.

Common failure modes

  • Watching the monthly invoice. By the time the invoice arrives, the runaway ran for a month. FinOps runs on near-real-time per-call data, not on the bill.
  • One shared API key. A single key gives you a total and no way to act on it. Attribution is the price of admission to governance.
  • Optimizing cost per token. It rewards cheap failures. Optimize cost per resolved task.
  • Ignoring the tail. The average cost per task looks fine while the p99 quietly funds a small yacht. The expensive tail is where the money and the fix both are.
  • Uncapped agent loops. No step limit and no token budget per task turns one stuck task into an open-ended charge.
  • Resending everything, every step. The most common agent cost bug is naive context growth. Summarize and prune instead of resending verbatim.
  • Forgetting caching. Teams pay full input price on prompts that are 90% identical call to call. Structure for cache hits and turn caching on.
  • Prompt changes with no cost review. A quiet context bloat in a prompt edit can outspend a quarter of roadmap work. Review prompt and model changes for cost.

A 30-day rollout

You do not need a program to start. A focused month gets you most of the value.

  • Week one, instrument. Route calls through a gateway or wrapper that logs model, tokens, cost, and tags on every call. Get one dashboard showing cost by feature over time.
  • Week two, attribute. Split the shared key into per-team or per-feature keys or tags. Publish the decomposition so every team sees its own spend.
  • Week three, optimize. Pull the top levers on your most expensive feature: trim context, add prompt caching, route easy calls to a cheaper model, cap output. Measure cost per resolved task before and after.
  • Week four, govern. Set hard caps per key, rate limits per user, and anomaly alerts. Give each budget an owner. Add a cost check to your prompt and model change review.

At the end of the month you can see spend, you know who owns it, you have cut the worst waste, and a runaway is a handled error instead of a surprise. That is a working FinOps practice, and everything after it is refinement.

FAQ

What is AI FinOps in one sentence? It is the practice of making AI token spend visible, attributed to teams and features, optimized for waste, and governed with budgets and hard limits, so the cost of running models is a managed number instead of a monthly surprise.

Why is token spend harder to manage than normal cloud spend? Because it is workload-dependent in a way instance-hours are not. An agent's cost depends on how hard each task turns out to be, which you do not know in advance, so spend is spiky and forecasting is harder. It also has its own pricing dimensions, input, output, cached, and context tier, that a generic cloud dashboard does not track.

Why do AI agents cost so much more than a chatbot? An agent makes many model calls per task instead of one, and it usually resends its growing context on every call, so tokens processed grow with roughly the square of the number of steps. One hard task that sends the agent down a long path can cost more than a hundred easy ones.

What is the single highest-leverage way to cut token spend? Trim the context you send, especially in agent loops, because context is resent on every step so every token you cut is saved many times. Close behind are routing easy calls to a cheaper model and caching stable prompt prefixes.

Should I optimize for cost per token? No. Optimize for cost per resolved task. A cheap model that fails and triggers retries or escalations can cost more to actually finish the work than a pricier model that succeeds once. Cost per resolution counts the failures against the outcome you wanted.

Do I need to buy a FinOps tool or can I build it? For one team with one main workload, build the per-call log, a dashboard, and hard caps yourself. It is a small project and it lives close to your code. Buy a cross-team attribution and reporting layer once several teams and providers are involved and maintaining the plumbing stops being worth it.

What stops a runaway loop from producing a huge bill? Hard caps per key and per feature that stop calls when a limit is hit, rate limits per user, a maximum step count and token budget per agent task, and anomaly alerts that flag abnormal spend within minutes. Dashboards report the disaster; caps prevent it.

How do I attribute spend when everything shares one API key? Split into a key or a propagated tag per team, product, environment, and where possible per feature and user. Capture the usage figures providers return on every response and record them with those tags. Attributed spend is what turns a single opaque total into something a team can own and reduce.