Training vs Inference: The Two Halves of AI

Every AI system you have ever used runs in two fundamentally different modes, and almost everything confusing about AI economics dissolves once you can tell them apart. Training is the process of learning: you feed a model enormous amounts of data and adjust billions of internal numbers until it gets good at predicting. It happens on someone else's schedule, costs a fortune, and — for any given model version — happens roughly once. Inference is the process of using: you send the finished model a prompt, it does a fixed amount of arithmetic, and it hands back an answer. It is cheap per call and paid on every single call, forever.

That asymmetry is the whole game. Training is a capital expense — a big upfront bet that a lab makes before anyone has typed a single prompt. Inference is an operating expense — a metered cost that scales with usage and never stops as long as the product is live. "The model is trained" is not the moment the bill stops. It is the moment the other bill starts.

Key takeaways

• Training learns the weights; inference uses them. Training runs the model backwards to update parameters; inference runs it forward to produce output. Same network, opposite direction.
• Training is (mostly) one-time; inference is per-query, forever. A model is trained once per version, then serves potentially trillions of queries. Over a popular model's life, cumulative inference cost dwarfs training cost.
• They stress hardware differently. Training is throughput-bound and memory-hungry (it must store gradients and optimizer state); inference is latency-sensitive and, for chat, often memory-bandwidth-bound.
• Per-token pricing hides which half you're paying for. API prices bundle a share of training amortization, but the marginal cost you drive is almost entirely inference.
• "Trained on the internet" and "running your query" are separate events. Privacy, freshness, and cost questions land differently depending on which half you mean.
• Fine-tuning is small training; RAG is smart inference. Knowing the split tells you which tool a problem actually needs.

Table of contents

• The one distinction that explains everything
• What training actually is, mechanically
• The three phases: pretraining, fine-tuning, post-training
• Training: the expensive, one-time(ish) half
• Inference: the cheap-per-call, pay-forever half
• What inference actually is: prefill and decode
• Why the memory and compute profiles diverge
• Why they use hardware so differently
• The batching asymmetry
• Why "training compute" is not your serving bill
• What the split means for the money
• Test-time compute: where reasoning blurs the line
• Where the line blurs (and where it doesn't)
• FAQ
• The bottom line

The one distinction that explains everything

A neural network is a giant mathematical function with billions of adjustable numbers called weights (or parameters). During training, those numbers start as noise and get nudged, batch by batch, toward values that make the model's predictions match the data. During inference, those numbers are frozen — the model just applies them.

The mechanical difference is direction. Training does a forward pass (make a prediction) and then a backward pass that computes how wrong the prediction was and how to adjust every weight to be slightly less wrong next time. That backward step — backpropagation — is where the learning lives and where most of the cost hides. Inference does the forward pass only. No backward pass, no weight updates, nothing remembered between requests.

This is why a running model does not "learn from you" mid-conversation. When a chatbot appears to remember what you said three messages ago, it is not updating weights — it is re-reading the whole conversation on every turn inside its context window. The weights set in training are identical for you and every other user. If you want the model to actually change, you have to run a training process again.

What training actually is, mechanically

It is worth slowing down on what "adjusting billions of numbers" concretely involves, because the mechanics explain nearly every cost difference that follows. A training step processes a batch of examples in three movements.

The forward pass. The model takes the batch and runs it through every layer to produce predictions — for a language model, a probability distribution over the next token at each position. This is arithmetically identical to what inference does. If training stopped here, it would cost roughly what inference costs. It does not stop here.

The loss. The predictions are compared against the known-correct answers using a loss function, which collapses "how wrong were we across this whole batch" into a single number. The entire goal of training is to make that number smaller.

The backward pass. This is the expensive, distinctive part, and it has no counterpart in inference. Using the chain rule of calculus, the model computes the gradient — the direction and magnitude by which each of its billions of weights should move to reduce the loss slightly. This is backpropagation, and it runs the network in reverse, layer by layer, from the loss back to the inputs. To do this, it must have kept every intermediate value ("activation") computed during the forward pass, because the gradient at each layer depends on them. That is why training memory balloons: inference can discard an activation the moment it is used, but training has to hold the whole forward pass in memory until the backward pass consumes it.

Then an optimizer applies the update. Modern training does not just subtract the raw gradient; it uses an optimizer like Adam that maintains, for every single weight, one or two additional running statistics (a smoothed gradient and a smoothed squared gradient). Those are the optimizer states, and they are why a model's training memory footprint is a large multiple of its parameter count. A rough mental model for mixed-precision training: for every parameter you must simultaneously hold the weight, its gradient, and two optimizer moments — before you have stored a single activation. That "many times the model's own size" figure is not hand-waving; it falls straight out of this arithmetic.

Repeat that three-movement cycle across trillions of tokens and you have a training run. Every cycle is a full forward pass plus a backward pass plus an optimizer update, on data that must be streamed through thousands of chips kept in lockstep. Inference, by contrast, is just the first movement, once, with everything downstream deleted.

The three phases: pretraining, fine-tuning, post-training

"Training" is not one event; it is a pipeline of distinct stages, and conflating them causes a lot of confusion about what a model "knows" and where its behavior comes from.

Pretraining is the enormous, undifferentiated stage: predict the next token across a vast corpus. This is where the megawatt-months and the headline dollar figures live. Pretraining buys raw capability — grammar, facts, reasoning patterns, code — but produces a model that is a talented text-completer, not a helpful assistant. It will happily continue a prompt in whatever direction the statistics point, including unhelpful or unsafe ones.

Fine-tuning adapts that base model to a narrower target. When you fine-tune a model, you run the exact same forward-plus-backward machinery described above, but on a small, curated dataset and usually for far fewer steps. It is categorically training — weights change — but it is measured in GPU-hours rather than GPU-months. Parameter-efficient methods (training a small set of adapter weights while freezing the rest) shrink the cost further, which is why fine-tuning is accessible to individual teams while pretraining is not.

Post-training is the umbrella for everything that turns a capable base model into a shippable product: instruction tuning, preference optimization (learning from human or AI comparisons of good versus bad answers), and safety alignment. This is where "be helpful, be honest, refuse the genuinely harmful thing" gets installed. It is still training in the mechanical sense — gradients and updates — but the data is behavioral rather than encyclopedic. Two models trained on nearly identical pretraining corpora can feel completely different because their post-training diverged.

Why does this taxonomy matter for the training/inference split? Because all three phases happen on the training side of the line — offline, before you ever send a prompt — and none of them recur when you query the finished model. When people ask "did my chat teach the model anything," the honest answer requires knowing that learning only happens inside one of these deliberate, offline stages, never during the live forward pass that answers you.

Training: the expensive, one-time(ish) half

Training a frontier model is one of the most capital-intensive things a software company can do. It requires thousands of accelerators wired together with fast interconnects, running for weeks or months, consuming megawatts of power. The cost is dominated by three things happening at once: the sheer number of arithmetic operations, the memory needed to hold not just the weights but also the gradients and optimizer state (often several times the size of the model itself), and the communication overhead of keeping thousands of chips in sync.

Crucially, training is bursty and finite. A lab decides to build a model, spends the money, and produces an artifact: a file of frozen weights. That artifact is the deliverable. Once it exists, the training cluster can move on to the next model. This is why we say training is "one-time" — per model version.

The "(ish)" matters, though. Training is one-time per version, and versions keep coming. Labs retrain, distill smaller models from big ones, and run fine-tuning passes that are themselves miniature training runs. There is also a legal and ethical dimension that lives entirely on the training side: what data went in. Debates about copyright and training data are debates about the training half — they concern what the model saw while learning, not what it does when you query it.

Inference: the cheap-per-call, pay-forever half

Inference is what happens when you actually use the model. You send tokens in, the model runs one forward pass per output token, and text comes back. For a language model, generation is autoregressive: it produces one token at a time, each new token requiring another pass over the network. This is why longer answers cost more and take longer — the work scales with how much text is produced.

Per call, inference is cheap. That is the seductive part. But it is paid on every call, by every user, every time, with no end date. A model that took a one-time fortune to train might serve billions of requests a month. Multiply a fraction of a cent by billions and the operating cost of running a popular model can, over its life, exceed what it cost to train it. The training bill is a headline; the inference bill is a subscription that never lapses while the product is live.

This is the single most important thing to internalize about AI economics: training is a sunk cost, inference is a recurring one. When you read that a model "cost X to train," that number says almost nothing about what it costs to keep serving. For a serious treatment of the serving side — GPU amortization, batching, when to self-host — see the inference cost economics guide. The mechanics of a single request are covered in how AI chatbots work.

What inference actually is: prefill and decode

"One forward pass per token" is true but hides a two-phase structure that dominates how inference performs and what it costs. A single request splits into prefill and decode, and they have almost opposite hardware personalities — the same asymmetry that shows up between training and inference reappears inside a single inference request.

Prefill processes your prompt. The whole input — every token of your question, your system prompt, and any retrieved context — is fed through the model in one shot. Because all those tokens are known up front, the math is a big, dense batch of matrix multiplications that saturates the accelerator's compute units. Prefill is compute-bound: it looks, briefly, a lot like a training forward pass. A long prompt makes prefill slower and is a real cost, which is one concrete reason a bloated context window is not free even before the model says anything.

Decode generates the answer, one token at a time. Here is the catch: each new token depends on all the previous ones, so the model cannot generate them in parallel. It produces token one, appends it, produces token two, and so on — an inherently sequential loop. On each step it does a tiny amount of math (a single token's worth) but must read the entire set of weights out of memory to do it. That makes decode memory-bandwidth-bound: the accelerator's arithmetic units sit mostly idle, starved, while the memory system shovels billions of weights across the bus for the sake of computing one token.

This prefill/decode divergence is important enough that serving stacks increasingly run the two phases on separate hardware pools tuned for each — the subject of disaggregated inference. It is also why the "time to first token" you feel (dominated by prefill) and the "tokens per second" you feel afterward (dominated by decode) are governed by different bottlenecks and improve for different reasons.

Why the memory and compute profiles diverge

Put the two sides beside each other and the resource asymmetry is stark, and it flows directly from the mechanics above rather than from any hardware mystique.

Training is compute-heavy and memory-heavy for the same reason: it remembers everything. It runs forward and backward, so it must retain activations, gradients, and optimizer state — collectively a large multiple of the raw parameter count. It is compute-bound because the backward pass roughly doubles the arithmetic of the forward pass, and it processes big batches to keep thousands of chips fed. Nobody minds if a single training step takes a full second, so latency is a non-issue; total throughput is everything.

Decode-phase inference is memory-bandwidth-bound because it remembers almost nothing but re-reads everything. There is no backward pass, no gradients, no optimizer state. The dominant memory cost is instead the KV cache: to avoid re-processing the entire conversation on every single generated token, the model stores the intermediate attention keys and values for all prior tokens and reuses them. That cache grows with the length of the conversation and the number of concurrent users, and it competes with the weights for scarce high-bandwidth memory. The full arithmetic of how the cache scales, and why it — not raw compute — is usually what caps how many users a GPU can serve at once, is worked through in the KV cache breakdown. The short version: in decode, you are not compute-limited, you are bandwidth-and-capacity-limited, and the KV cache is the reason.

So the same physical accelerator, running the same weights, presents as two different machines. During training it is a compute furnace that hoards memory to enable learning. During inference decode it is a memory-bandwidth pump whose arithmetic units are half-asleep. That single fact — that the bottleneck moves depending on which half you are running — underlies almost every serving optimization in the field.

Why they use hardware so differently

People assume training and inference just need "GPUs," but the two workloads pull hardware in different directions.

Training wants raw throughput and lots of memory. It processes huge batches of examples in parallel, and it must store gradients and optimizer state alongside the weights — often 3-4x the model's own size in memory. It tolerates high latency (nobody cares if a training step takes a second) but is desperate for total floating-point throughput and fast chip-to-chip networking, because thousands of accelerators have to stay synchronized.

Inference wants low latency and high memory bandwidth. When you generate one token at a time, you are not doing much math per token — you are mostly reading the weights out of memory to apply them. That makes single-user chat generation memory-bandwidth-bound rather than compute-bound. Serving systems fight back by batching many users' requests together so the expensive weight-read is shared, and by caching intermediate state so the model doesn't recompute the whole conversation each turn.

Dimension	Training	Inference
Direction	Forward + backward pass	Forward pass only
Weights	Being updated	Frozen
Frequency	Once per model version	Every single query
Cost type	Capital expense (upfront)	Operating expense (recurring)
Bottleneck	Compute throughput + interconnect	Latency + memory bandwidth
Memory needs	Weights + gradients + optimizer state	Weights + running context (KV cache)
Time scale	Weeks to months	Milliseconds to seconds
Who pays	The lab, before launch	Everyone, on every call

The batching asymmetry

Batching — running many examples through the model at once — is central to both halves, but it plays opposite roles, and understanding why is the key to understanding inference cost.

In training, batching is a straightforward win with a knob you fully control. You are processing a fixed dataset offline, so you gather as many examples as memory allows into each step. A larger batch means the fixed cost of reading the weights is amortized over more examples, and the gradient estimate is less noisy. There is no user waiting, so you push the batch as large as the hardware and the optimization math tolerate. Batching in training is a design decision made once.

In inference, batching is a win you have to steal from latency in real time. Recall that decode is memory-bandwidth-bound: reading the weights to generate one token for one user costs almost the same as reading them to generate one token for fifty users simultaneously, because the expensive part — hauling the weights across the memory bus — is shared. So batching many users together is how a provider turns a hopelessly inefficient single-user workload into an economical one. The catch is that the users do not arrive in a neat batch. They show up at random, mid-conversation, with prompts of wildly different lengths, and each one is waiting for a response now. Serving systems therefore do continuous batching: they dynamically slot new requests into a running batch and evict finished ones token by token, rather than waiting to assemble a fixed group.

This is the asymmetry that most surprises people: an idle GPU serving one user is enormously wasteful, because it pays the full weight-read cost for a single token of output. The economics of inference only work at scale and concurrency. A model serving thousands of simultaneous users can be efficient; the same model serving one user per second is burning money. It is the mirror image of training, where a single job saturates the hardware by construction. This is also why self-hosting a large model for low, spiky traffic so often loses to an API: you pay for the whole accelerator but cannot fill the batch that would make it cheap.

Why "training compute" is not your serving bill

When a model launches, the number that makes headlines is the training compute — some vast figure of floating-point operations or dollars. It is a real number and a real barrier to entry. It is also almost useless for predicting what it costs to use the model, and conflating the two is one of the most common errors in AI commentary.

Here is the disconnect. Training compute is spent once, up front, to produce the weights, and it scales with the size of the model and the amount of data it learned from. Serving cost is spent per token, forever, and it scales with how many people use the model and how much they generate. These two quantities are not proportional. A lab can spend a fortune pretraining a model and then, through distillation and quantization, serve a much smaller, cheaper derivative that captures most of the capability. The expensive training run and the cheap served model can be different artifacts entirely.

Two models with identical training compute can have wildly different serving costs, driven by architecture rather than training budget. A dense model reads all its weights for every token; a mixture-of-experts model of the same nominal size routes each token through only a fraction of its parameters, so it serves far more cheaply despite a comparable or larger training bill. Meanwhile a "reasoning" model with a modest training cost can be expensive to serve because it generates enormous chains of intermediate tokens per answer. The lesson holds in both directions: the headline training number tells you what it took to build the model, and says almost nothing about what a token of output will cost you. For the figures that actually determine serving economics — utilization, batching, GPU amortization, and when self-hosting pays off — the inference cost economics guide does the arithmetic. On GPUs specifically, note that the chips optimized for training (maximum interconnect and memory to hold optimizer state) are frequently not the most cost-effective chips for serving, which rewards memory bandwidth over raw training throughput — another reason the two bills decouple.

What the split means for the money

Once you see AI as two cost centers, a lot of industry behavior stops looking mysterious.

Why frontier models come from a handful of players. Training is a capital barrier. Only organizations that can spend enormous sums upfront get to produce frontier weights. This is also why open-weights releases matter so much: when a lab publishes trained weights, it hands the world the expensive half for free, leaving only the cheaper inference cost for everyone else to pay.

Why API prices look low but add up. A provider prices per token to recover a slice of training amortization plus the real marginal cost of serving. The marginal cost you actually drive is almost entirely inference. Prompt for a shorter answer and your bill drops — because you cut inference work, not training. Learning to write efficient prompts is, in cost terms, learning to buy less inference per useful result.

Why "reasoning" models cost more to run. Models that "think" before answering generate large amounts of intermediate tokens. Those tokens are pure inference — every one is another forward pass. The training didn't get more expensive; the usage did. This is the training/inference split showing up as a line item.

Why the environmental story is two stories. Training's footprint is a big, discrete pulse of energy tied to building each model. Inference's footprint is a steady drip that scales with adoption. For a rarely-used model, training dominates. For a wildly popular one, the accumulated inference energy can overtake the training pulse many times over. Any honest AI energy and water footprint claim has to say which half it is counting.

Test-time compute: where reasoning blurs the line

For years the split was clean: capability was purchased during training, then spent cheaply at inference. Reasoning models complicate that story in an interesting way, and it is worth being precise about how, because the popular framing ("the model thinks at inference time") is easy to over-read.

A reasoning model is trained — through post-training on examples of working through problems step by step — to spend more inference tokens before committing to an answer. It generates a long internal chain of intermediate reasoning, then a final response. This is often called test-time compute: you can make the model's answers better not by training a bigger model, but by letting the deployed model do more inference on each query. Spend more decode tokens, get a better answer. Capability that used to come only from a larger training run can now, partly, be bought at serving time.

Does this erase the training/inference distinction? No — it sharpens it, and here is the crucial nuance. The reasoning behavior was still installed during training; a base model does not spontaneously produce useful chains of thought. What test-time compute shifts is where the marginal capability comes from. Under the old regime, a fixed inference budget per query gave you whatever the training baked in. Under the new one, you have a dial at inference time — think longer, pay more per answer, get better results — and every extra "thinking" token is pure inference: another sequential, memory-bandwidth-bound decode step, another line on the operating bill. Nothing about the training got more expensive when you turned the dial up; your usage did.

The economic consequence is direct and often underappreciated. Reasoning models can generate five or ten times the tokens of a direct answer, and because those tokens are decode tokens, they carry the full sequential cost. This is why reasoning modes are metered and gated: they trade money-per-query for quality, moving spend firmly onto the inference side of the ledger. The serving challenges specific to these long-generation workloads — and why they strain batching and KV-cache capacity in particular — are covered in reasoning model serving. The clean summary: reasoning did not blur the line between the two halves so much as reveal that you can now buy capability on either side of it, and that the inference side, being recurring, is the one that quietly compounds.

Where the line blurs (and where it doesn't)

The clean split has real edge cases worth naming, precisely because they are so often confused.

Fine-tuning is training, just smaller. When you fine-tune a model, you run a genuine training process — forward and backward passes, weight updates — on a modest dataset. It is cheaper than pretraining but it is categorically training: it produces new weights.

RAG is inference, just smarter. Retrieval-augmented generation makes a model appear to "know" new facts without any training. It does this purely at inference time by fetching relevant text — often via vector search — and stuffing it into the context window before the forward pass. No weights change. This is why RAG is the go-to for keeping answers current: retraining is slow and expensive; retrieval is a cheap inference-time trick. If you are choosing between them for a product, the split is the decision framework — see how to choose an LLM for your app.

Memory features are inference, not learning. When an assistant "remembers" you across sessions, it is almost always storing notes in a database and re-injecting them into the context on later calls. The weights never move. This distinction matters for privacy: "was my chat used to train the model" (a training question) is very different from "is my chat stored and replayed to me later" (an inference/storage question).

Where it genuinely does not blur: a deployed model serving your query is not learning from it. Full stop. Any product that truly learns from interactions is running a separate training pipeline in the background on collected data — a distinct, deliberate, offline process.

FAQ

Does an AI model learn from my conversations while I chat with it? No. During inference the weights are frozen; the model does not update itself from your messages. Apparent "memory" comes from re-reading the conversation each turn or from stored notes injected back into the prompt. Real learning requires a separate training run, done offline on collected data — not live during your chat.

If training is a one-time cost, why do AI companies keep spending so much? Two reasons. First, training is one-time per model version, and new versions ship constantly. Second, and more importantly, inference never stops: every query on a live product costs money to serve. For a popular model, cumulative inference spend over its lifetime can exceed its training cost. The bill doesn't end when training does — it changes shape.

Which costs more overall, training or inference? It depends on usage. For a model that is rarely queried, the upfront training cost dominates. For a widely used model serving billions of requests, accumulated inference cost typically overtakes training cost, often many times over. Training is the headline number; inference is the recurring one that quietly gets larger.

Why is inference sometimes slow if the model is already trained? Because inference still does real work per request: language models generate one token at a time, each requiring a full forward pass over billions of weights, and each pass is limited by how fast those weights can be read from memory. Longer answers and longer contexts mean more passes and more data movement — hence more latency. Being "trained" removes the learning step, not the computing step.

Is fine-tuning training or inference? Fine-tuning is training — a smaller, cheaper training run that updates weights on a focused dataset. If you want to add knowledge without changing weights, that is a job for retrieval (RAG), which happens entirely at inference time. A good rule: if weights change, it's training; if only the prompt changes, it's inference.

Do training and inference need the same hardware? They can run on the same accelerators, but they favor different traits. Training rewards raw compute throughput, large memory (for gradients and optimizer state), and fast interconnects between many chips. Inference rewards low latency and high memory bandwidth, and is usually served in batches to share the cost of reading the weights. Many providers now use different, cheaper configurations for serving than for training.

Why does training need so much more memory than inference for the same model? Because training has to remember its work in order to learn from it. The backward pass computes how to adjust each weight, and that calculation needs every intermediate value from the forward pass (the activations), plus a gradient for every weight, plus optimizer state (typically two extra running statistics per weight). Stacked together, that is several times the model's own size before you count the batch. Inference runs forward only, discards each activation as soon as it is used, and keeps no gradients or optimizer state — its main extra memory cost is the KV cache for the current conversations, not the machinery of learning.

What are prefill and decode, and why do they behave differently? They are the two phases of a single inference request. Prefill processes your whole prompt at once; because all tokens are known, it is a dense, compute-bound burst that governs how long you wait for the first token. Decode then generates the answer one token at a time, sequentially, reading all the model's weights from memory on each step — that makes it memory-bandwidth-bound and governs your tokens-per-second. Different bottlenecks, which is why they are increasingly served on separately tuned hardware.

Do reasoning models cost more because they were trained differently or run differently? Both, but the recurring cost is on the running side. The reasoning behavior is installed during post-training, a one-time expense. What you pay repeatedly is the inference: a reasoning model generates a long chain of intermediate "thinking" tokens before its answer, and every one of those is another sequential decode step on the operating bill. Turning up the "think longer" dial raises your usage cost, not the model's training cost.

The bottom line

Training and inference are not two features of AI — they are its two economies. One is a capital bet a lab makes before you show up: expensive, finite, and where questions of data and capability are settled. The other is the meter that runs every time anyone uses the result: cheap per tick, relentless in aggregate, and where questions of cost, latency, and privacy actually live.

Keep the split in your head and the rest of the field gets legible. You will know why open weights are such a gift, why a shorter prompt saves money, why a model can't answer questions about last week without retrieval, and why "we finished training" is a starting line, not a finish. If you want the broader map of how these pieces fit together, the AI canon is the place to start; for where this is all heading, see AI over the next 10 years.