What Is a GPU, and Why Does AI Need Them?

A GPU — graphics processing unit — is a chip designed to do thousands of simple arithmetic operations at the same time. It was built to paint millions of pixels per frame in a video game, where every pixel can be computed independently. That single design choice — trade the ability to do one thing very fast for the ability to do many things at once — is exactly what a neural network needs, because the core of every neural network is a pile of multiplications that don't depend on each other. AI runs on GPUs not because of marketing or momentum, but because the math of AI and the architecture of a GPU happen to be the same shape.

Here is the part most explanations skip. People assume the GPU wins because it does more math per second. That's half the story, and the less important half. The real constraint in modern AI isn't how fast a chip can multiply — it's how fast it can feed the multipliers with numbers from memory. Understanding that one inversion — that memory bandwidth, not raw compute, is usually the ceiling — is the difference between parroting "AI needs GPUs" and actually understanding why AI is expensive, why it's sometimes slow, and why the chips are perpetually scarce.

Key takeaways

• A GPU is a parallelism machine. It has thousands of simple cores that do arithmetic simultaneously, versus a CPU's handful of complex cores optimized to finish one task as fast as possible.
• The whole game is matrix multiplication. Neural networks are, mechanically, sequences of matrix multiplies. Every one is a mountain of independent multiply-then-add operations — the exact workload a GPU exists to crush.
• Memory bandwidth is the real bottleneck, not FLOPs. Modern accelerators can multiply far faster than they can pull numbers from memory. Most of the time the arithmetic units are idle, waiting on data.
• "GPU" is now a loose label. The chips powering AI have specialized cores (tensor cores), high-bandwidth memory stacked next to the compute, and interconnects to gang thousands of them together. They're AI accelerators that kept the name.
• This one architecture explains the economics. Cost, latency, and scarcity all trace back to parallelism and bandwidth. If you only learn one hardware concept, learn this one.

Table of contents

• Key takeaways
• The CPU vs GPU split: latency vs throughput
• Inside the machine: SIMT, warps, and why branches hurt
• Why matrix multiplication is the entire job
• The plot twist: memory bandwidth is the real ceiling
• Arithmetic intensity and the roofline
• Tensor cores and the precision ladder
• What "a GPU" even means now
• Training vs inference: two different hungers for the same silicon
• Why one GPU is never enough: parallelism across chips
• Datacenter silicon vs the card in your gaming PC
• The software moat: CUDA and why the ecosystem is the product
• How much GPU do you actually need?
• Why this one idea explains the whole industry
• FAQ
• The one-sentence version

The CPU vs GPU split: latency vs throughput

Start with the CPU, because the GPU is defined against it.

A CPU is a latency machine. Its job is to take one instruction stream — the logic of a program, full of branches, decisions, and dependencies — and get through it as fast as physically possible. To do that it throws enormous resources at making a single thread quick: deep instruction pipelines, branch prediction, out-of-order execution, large caches, high clock speeds. A modern CPU might have somewhere between a handful and a few dozen cores, each one a sophisticated general-purpose engine that can do almost anything. It is a small team of brilliant generalists.

A GPU is a throughput machine. It doesn't care how long any single operation takes. It cares about total work completed per second across a massive batch. So instead of a few clever cores, it packs in thousands of simple ones. Each individual core is slower and dumber than a CPU core — worse at branching, worse at anything with unpredictable control flow. But there are thousands of them, and when your problem is "do this same arithmetic to a million different numbers," thousands of dumb cores obliterate a few dozen smart ones. It is a giant army of specialists who all do the exact same drill.

The classic analogy: a CPU is a sports car — one passenger, gets there fast. A GPU is a bus — slow to fill and slow off the line, but it moves hundreds of passengers per trip. If your task is "move one person quickly," the car wins every time. If your task is "move a stadium," you want buses, and it isn't close.

	CPU	GPU
Design goal	Finish one task with minimum delay (latency)	Finish maximum work per second (throughput)
Cores	Few, complex, general-purpose	Thousands, simple, specialized
Best at	Branchy, sequential logic; running an OS	Doing the same operation to huge data in parallel
Weakness	Can't parallelize wide enough	Terrible at unpredictable, branch-heavy code
Analogy	Sports car	Bus

Neither is "better." They're answers to different questions. It just happens that the question AI asks — do this enormous batch of identical, independent arithmetic — is the exact question the GPU was built to answer.

There's a subtler point buried in that table, and it's worth dragging into the light because it explains a lot of downstream behavior. The CPU spends most of its transistor budget on not doing arithmetic. Branch predictors, reorder buffers, speculative execution, multiple layers of cache — these are all machinery for keeping a single instruction stream fed and moving despite the fact that real programs are full of unpredictable turns. That machinery is expensive in silicon area and power, and it exists purely to fight latency. The GPU makes the opposite bet: it strips almost all of that away and spends the reclaimed transistors on more arithmetic units. It can afford to, because it hides latency a completely different way — not by predicting what comes next, but by having so much independent work queued up that it can always switch to something else while it waits. Hold that idea; it's the key to the next section.

Inside the machine: SIMT, warps, and why branches hurt

"Thousands of cores" is a useful lie. It's close enough for the mental model, but the way a GPU actually organizes those cores explains both its superpower and its Achilles' heel, and once you see it, several otherwise-mysterious behaviors snap into focus.

A CPU core executes its own instruction stream — SIMD extensions aside, each core decides independently what to do next. A GPU doesn't work that way. Its cores are herded into groups (NVIDIA calls a group of 32 a warp; other vendors use different names and sizes), and every core in a group executes the same instruction at the same time, just on different data. This is the SIMT model — Single Instruction, Multiple Threads. One instruction is fetched and decoded once, then applied across the whole group in lockstep. That's why the arithmetic is so cheap: you amortize the cost of fetching and decoding an instruction across dozens of data elements. The control logic that a CPU replicates per-core is shared across the whole warp on a GPU.

This is also why GPUs are terrible at branchy code, and it's not a vague "they don't like branches" — there's a precise mechanism. Suppose an if statement sends half the threads in a warp down one path and half down another. Because the whole warp must execute one instruction at a time, the hardware can't run both paths simultaneously. It runs the first path with the "else" threads switched off (idle), then runs the second path with the "if" threads switched off. The two halves are serialized. This is called warp divergence, and it can halve your throughput on a single branch, quarter it on a nested one. The lesson: a GPU's parallelism is real only when every lane is doing the same thing. Matrix multiply obliges perfectly — every element runs identical multiply-and-add code with no branches — which is one more reason the workload and the machine fit like a key in a lock.

Now the latency-hiding trick from the last section. A GPU core, having issued a memory request, does not sit and wait for the number to arrive. Instead the hardware instantly swaps in a different warp that has its data ready and runs that one, then another, cycling through a large pool of resident warps. By the time it comes back around, the original request has (hopefully) landed. The GPU keeps its arithmetic units busy not by making any single memory access fast — it can't — but by having so many independent threads in flight that there's always someone ready to run. This is latency hiding through massive multithreading, and it's the deepest architectural difference from a CPU. The CPU fights latency with prediction and caching; the GPU drowns it in parallelism. It's an elegant answer, but notice the catch that will haunt the rest of this article: the trick only works if there's enough independent work and enough memory bandwidth to service all those outstanding requests. Run short on either and the whole scheme stalls.

Why matrix multiplication is the entire job

Here's the claim that makes everything click: running a neural network is, mechanically, doing a long series of matrix multiplications. Not "involves" matrix multiplication as one step among many. It is matrix multiplication, over and over, with some cheap nonlinear functions sprinkled between. If you understand why matrix multiply loves a GPU, you understand why AI needs one.

A neural network layer takes a list of numbers (your input), and produces a new list of numbers (the output passed to the next layer). The way it does this is: every output number is a weighted sum of every input number. Stack many outputs and many inputs together and that operation is, by definition, a matrix multiply — a grid of weights times a vector of inputs. The transformer architecture behind modern language models is stacks of exactly these operations: the attention mechanism is matrix multiplies, the feed-forward layers are matrix multiplies. The model's "knowledge" is billions of weights, and using it means multiplying your input through all of them.

Now here's the property that matters. To compute output number 1, you multiply the inputs by the first row of weights and add them up. To compute output number 2, you use the second row. Output 1 and output 2 do not depend on each other. You could compute all of them at literally the same instant if you had enough hands. A matrix multiply of a 1,000-wide layer is thousands of these little multiply-and-add jobs, every one independent of the rest.

That is a parallelism machine's dream. Hand each of a GPU's thousands of cores a chunk of the multiply-and-adds, run them all at once, collect the results. A CPU, with its few cores, has to march through them in small groups — fast per group, but there are millions of groups. The GPU eats the whole thing in a fraction of the wall-clock time.

This is also why training and inference cost what they do — two workloads that stress this hardware very differently, as training vs inference lays out. A large model is billions of weights. Generating even a single token means pushing numbers through all of them — billions of multiply-and-adds. Do that for every token, for every user, and you see why the arithmetic has to be parallel or it would be hopeless. The GPU didn't make AI possible by inventing new math. It made the existing math cheap enough to do at planetary scale.

The plot twist: memory bandwidth is the real ceiling

Now the part that separates a real understanding from a shallow one.

The intuitive story is: AI needs to do a colossal amount of arithmetic, GPUs do arithmetic fast, done. And people reach for the headline spec — FLOPS, floating-point operations per second — as the measure of a chip. More FLOPS, more better.

Except in practice, the arithmetic units on a modern AI chip spend most of their time idle, waiting for numbers to arrive. The multipliers are not the bottleneck. Getting data to them is.

Think about what a multiply-and-add actually requires. Before the core can multiply two numbers, those two numbers have to physically travel from memory into the core. On modern hardware, the multiply itself is almost free — the arithmetic units are absurdly fast. The expensive, slow, energy-hungry part is the trip: fetching the weight and the input value out of memory and moving them to where the compute happens. A chip can often do dozens of arithmetic operations in the time it takes to fetch a single number from memory. So if every number you fetch only gets used once, your thousands of cores sit around bored while the memory system heaves data at them as fast as it can — which is never fast enough.

This is why memory bandwidth — the rate at which a chip can move data between its memory and its compute cores, not the rate at which it can multiply — is usually the real ceiling for AI. When a language model generates text one token at a time, it has to read a large fraction of its billions of weights out of memory for every single token, and each weight gets used just once before it's discarded. There's almost no arithmetic to hide the wait behind. The generation speed you experience is set almost entirely by how fast the model's weights can be streamed out of memory. The multipliers are barely breaking a sweat.

Engineers have a name for this: a workload is memory-bound when it's limited by data movement, and compute-bound when it's limited by arithmetic. A huge amount of AI — especially the token-by-token generation you hit when you use a chatbot — is memory-bound. This is the single most important and least understood fact about AI hardware. It reframes everything:

• Why AI chips carry exotic memory. The premium accelerators bolt ultra-fast memory (high-bandwidth memory, HBM — stacked right next to the compute) onto the die. You're not mostly paying for more multipliers. You're paying for a wider, faster pipe to feed them.
• Why batching makes AI cheaper. Serve many users' requests together and each weight you fetch from memory gets reused across all of them — one expensive trip, many multiplies. You've converted a memory-bound job into a compute-bound one and finally put those idle multipliers to work. It's the core trick of efficient AI serving.
• Why a bigger context window hurts. More context means more intermediate data to store and stream, and the memory system — already the bottleneck — takes on more load.
• Why quantization is such a big deal. Storing weights in smaller number formats means fewer bytes to move per weight. Since you're bandwidth-limited, shrinking the data moves the ceiling directly. It's a memory-bandwidth optimization first, a storage saving second.

If you only remember one thing past "GPUs do parallel math," remember this: the bottleneck is usually feeding the math, not doing it. Chase FLOPS and you'll misprice, mis-provision, and misunderstand every hardware decision downstream.

Arithmetic intensity and the roofline

There's a clean way to make "memory-bound versus compute-bound" precise instead of hand-wavy, and it's worth internalizing because it turns a fuzzy intuition into a number you can reason with. The concept is arithmetic intensity: for a given piece of work, how many arithmetic operations do you perform for each byte you move from memory? It's a ratio — FLOPs per byte. Low intensity means you touch a lot of data and do little math with it (memory-bound). High intensity means you do a lot of math on each byte you fetch (compute-bound).

Every chip has a matching ratio baked into its hardware: divide its peak arithmetic rate by its peak memory bandwidth and you get the intensity at which the two are balanced — the point where, in principle, the multipliers and the memory pipe finish at the same instant. Below that ratio, your work is bottlenecked by bandwidth and the expensive arithmetic units sit partly idle no matter how many FLOPS the spec sheet advertises. Above it, you're finally limited by compute. This is the roofline model, and if you sketch it — a rising diagonal line (the bandwidth limit) that flattens into a horizontal ceiling (the compute limit) — you can literally see where any workload lands and which resource is holding it back.

Here's why this matters for AI specifically. The two big operations in a transformer sit on opposite sides of that line. A large matrix multiply where you multiply a big weight matrix by a big batch of inputs has high arithmetic intensity — each weight you fetch gets reused across every row of the batch, so you extract many FLOPs per byte. That work is compute-bound and lives up under the flat roof, using the hardware well. But the token-by-token generation phase, where the batch is effectively one, is the opposite: each weight is fetched and used essentially once. Intensity collapses toward one operation per byte, the work slides down the diagonal, and you're pinned against the bandwidth wall with most of your multipliers unemployed.

This single framework explains why nearly every serving optimization is, at bottom, an effort to raise arithmetic intensity — to climb up the diagonal toward the roof:

• Batching stacks many requests so each fetched weight is reused across all of them. More FLOPs per byte. It literally moves you rightward on the roofline into compute-bound territory.
• KV caching avoids re-reading and recomputing past tokens' work on every step, cutting wasted data movement.
• Quantization shrinks the bytes-per-weight denominator, so even at fixed math you move fewer bytes and your effective intensity rises.
• Fusing operations keeps intermediate results in fast on-chip memory instead of writing them out to main memory and reading them back — you do more math per round trip.

None of these add multipliers. They all rearrange the work so the multipliers you already paid for actually run. When you read that a team "improved GPU utilization," this is almost always what happened underneath: they hauled a memory-bound workload up the roofline toward its compute ceiling.

Tensor cores and the precision ladder

If matrix multiply is the whole job, the obvious hardware move is to stop doing it with general-purpose parallel cores and build silicon that does nothing but small matrix multiplies. That's what a tensor core (NVIDIA's name; other vendors ship "matrix engines," "matrix multiply units," and the like) is. A general GPU core multiplies two numbers and adds a third per instruction. A tensor core swallows two small matrices in one shot and spits out their full product — a whole tile of multiply-and-adds fused into a single hardware operation. Since the transformer is a tower of matrix multiplies, dedicating a large fraction of the chip's transistors to this one operation pays off enormously, and it's a big reason modern accelerators post arithmetic rates that dwarf what the general cores alone could reach.

But tensor cores come bundled with a second idea that's just as important: lower numerical precision. To understand why, remember that every number in a neural network is stored in some floating-point format, and the format's bit-width sets both how much memory each number occupies and how fast the hardware can crunch it. Traditional scientific computing uses 32-bit or 64-bit floats because it needs precision. Neural networks, it turns out, don't — they're statistical, noise-tolerant systems, and a slightly imprecise weight is no worse than a slightly different weight the training process would have accepted anyway. So the industry has walked steadily down a precision ladder: 32-bit, then 16-bit formats (FP16 and BF16, which trade precision bits for a wider range to keep training stable), then 8-bit (FP8), and increasingly formats narrower still.

Every rung down the ladder buys two things at once, and both matter because of everything above. Fewer bits per number means more matrix multiplies per second — a tensor core fed 8-bit inputs can process far more of them than the same silicon fed 16-bit inputs, because the multiply hardware is simpler and you can pack more of it in. And fewer bits per number means fewer bytes to move — which, as the roofline made painfully clear, is the actual bottleneck most of the time. Lower precision attacks both the compute ceiling and the bandwidth wall simultaneously, which is exactly why it's one of the highest-leverage moves in all of AI hardware and why every new accelerator generation reaches for a narrower format.

The catch is that you can't shrink precision for free forever; push too far and the model's quality degrades or its training destabilizes. Choosing how many bits to spend, and where, is a genuine engineering tradeoff with real failure modes — which is a whole subject of its own, covered in the quantization tradeoffs guide. For the hardware picture, the thing to hold is the shape: specialized matrix-multiply units, fed the narrowest numbers the model can tolerate, is the second great lever after parallelism itself.

What "a GPU" even means now

The chips running frontier AI are, strictly, barely graphics processors anymore. Many don't even have display outputs. They kept the name for historical reasons, but calling them a GPU is like calling a modern smartphone a "telephone" — technically descended from one, functionally a different animal. The honest term is AI accelerator. Here's what actually distinguishes the current generation:

• Specialized matrix-multiply cores. On top of the general parallel cores, modern accelerators include hardware blocks built to do nothing but small matrix multiplies at high speed — often marketed as "tensor cores" or similar. Since matrix multiply is the whole workload, dedicating silicon to it pays off enormously. (Naming varies by vendor and generation; treat any specific brand name as a snapshot in time.)
• High-bandwidth memory glued to the compute. As of writing, the defining feature of a top-tier AI chip isn't its arithmetic rate — it's the stack of HBM sitting millimeters from the cores, feeding them. This is a direct answer to the memory-bandwidth ceiling.
• Interconnects to gang chips together. The largest models don't fit on one chip. Their weights are split across many accelerators wired together with high-speed links, so a rack behaves like one enormous GPU. When people talk about clusters of thousands of GPUs training a model, the interconnect is what makes them cooperate instead of just sitting in the same building.
• Lower-precision number formats. These chips are built to compute in small formats (8-bit, or even smaller) because AI tolerates imprecise arithmetic surprisingly well — and, as established, fewer bits means less to move.

The specific product names, memory sizes, and core counts will churn constantly. Don't anchor to them. Anchor to the shape: massively parallel arithmetic, fed by the fastest possible memory, ganged together by fast interconnects. That shape is stable. The model numbers on top of it are not.

It's also worth naming the alternatives, because "GPU" isn't the only accelerator. Some operators build custom AI chips (TPUs and other ASICs) that hard-wire the same ideas even more aggressively — less general-purpose flexibility, more matrix-multiply-and-bandwidth per dollar. They're not fundamentally different in concept. They're the same bet — parallel matmul, fed fast — with different tradeoffs on how specialized to go. The GPU just got there first because gaming had already paid to build the parallel-hardware ecosystem. A dedicated ASIC can win on efficiency for the exact workload it was designed for, but it pays for that with rigidity: when the shape of the models shifts — a new attention variant, a new number format — a flexible GPU can often just run it, while fixed-function silicon may need a new tape-out that takes years. That tension between efficiency and flexibility is the whole story of AI hardware competition, and there's no permanent winner, only a moving frontier.

Training vs inference: two different hungers for the same silicon

People say "you need GPUs for AI" as if there were one workload. There are two, and they stress the hardware so differently that the same chip can be the right tool for one and the wrong tool for the other. The full comparison lives in training vs inference; here's the part that's specifically about the metal.

Training is teaching the model — showing it mountains of data and adjusting billions of weights until it gets good. Mechanically it runs each batch forward to make a prediction, then backward to compute how every weight should change, then applies the update. That backward pass is where the hardware demands explode. To compute the corrections you must remember the intermediate results of the forward pass, so memory capacity fills up with activations, gradients, and optimizer state — often several times the size of the model itself. Training also runs on huge batches by design, which, per the roofline, is exactly the high-arithmetic-intensity regime where the tensor cores run hot and compute-bound. Training is where raw FLOPS genuinely earn their keep, and where you need not one accelerator but a coordinated army of them running for weeks.

Inference is using the finished model — one forward pass, no backward, no weight updates. It splits into two phases with opposite personalities. The prefill phase (digesting your prompt) processes all the input tokens at once, so it's parallel, high-intensity, and compute-bound — it behaves a bit like training. The decode phase (generating the answer one token at a time) is the memory-bound nightmare from earlier: each new token requires streaming a large fraction of the weights from memory to produce a single result, with nothing to batch against unless you're serving many users together. This is why the two phases feel different in practice — a long prompt with a short answer stresses compute; a short prompt with a long answer stresses bandwidth.

The practical upshot: a chip loaded with arithmetic units but modest memory can be a training beast and an inference disappointment, or the reverse. Inference at scale is often more about memory capacity (to hold the model plus every concurrent user's context) and bandwidth (to stream weights fast) than about peak FLOPS. This is why the industry increasingly designs and buys different hardware for the two jobs, and why "how many GPUs does it take to run this model" has no single answer until you say which job you mean.

Why one GPU is never enough: parallelism across chips

A frontier model does not fit on one accelerator. Its weights alone can run to hundreds of gigabytes, dwarfing the memory on any single chip, and even when a model does fit, you often want many chips working on it to go faster. So the real unit of modern AI hardware isn't a GPU — it's a cluster of them lashed together, and the wiring between them is as important as the chips themselves.

The problem is that splitting a neural network across chips forces them to talk, constantly, because the layers depend on each other's outputs. That communication becomes a bottleneck of its own, so the industry has evolved several ways to split the work, each with a different communication pattern:

• Data parallelism: give every chip a full copy of the model but a different slice of the training batch. They compute independently, then must synchronize their weight updates every step — a big, regular exchange of gradients.
• Tensor parallelism: split a single layer's matrix multiply across several chips, each doing part of the multiply. This demands extremely fast, low-latency chatter within every layer, which is why it's reserved for chips wired together most tightly.
• Pipeline parallelism: put different layers on different chips, so data flows through them like an assembly line. Less frequent communication, but you have to keep the pipeline full or chips sit idle waiting for work.

Real systems combine all three, and the reason the wiring matters so much is that if the chips can't exchange data fast enough, they stall — you've bought a thousand accelerators and they spend their time waiting on each other instead of computing. So datacenters build a hierarchy of interconnect. Inside a server, a handful of accelerators are joined by an ultra-fast, short-range fabric (NVIDIA's is branded NVLink) that lets them behave almost like one giant chip with pooled memory. Across servers, racks are stitched together with high-speed networking (typically InfiniBand or specialized Ethernet) so thousands of chips form one training system. The whole point is to make a rack — or a room — behave like a single enormous GPU. When people quote clusters of tens of thousands of accelerators, the interconnect is the unsung hero that turns a pile of chips into a coherent machine, and it's frequently the harder engineering problem than the chips themselves.

Datacenter silicon vs the card in your gaming PC

It's tempting to think a datacenter AI chip is just a bigger version of the gaming card in a desktop. They share DNA, and a high-end consumer card genuinely can run smaller models. But the differences are exactly the things this article has been building toward, which makes them a useful review. (For a concrete tour of who makes what, see the current datacenter GPU lineup.)

• Memory type and capacity. This is the big one. Consumer cards use fast graphics memory (GDDR); datacenter accelerators use HBM — high-bandwidth memory stacked in three dimensions and bonded millimeters from the compute, delivering several times the bandwidth. Since so much of AI is bandwidth-bound, this alone separates the two classes. Datacenter parts also carry far more memory capacity, because they have to hold enormous models and many users' context at once.
• Interconnect. Consumer cards mostly talk to the rest of the computer over the standard PCIe slot and have no fast way to gang together. Datacenter parts add dedicated chip-to-chip links (NVLink and friends) precisely so they can form the clusters described above. You can't build a serious training system out of gaming cards mainly because they can't be wired together tightly enough.
• Reliability features. A training run lasting weeks across thousands of chips cannot tolerate silent data corruption, so datacenter memory uses ECC (error-correcting code) to catch and fix bit flips. Consumer parts often skip it. When one corrupted number can poison a multi-week, multi-million-dollar run, this stops being a nicety.
• Precision and features. Datacenter chips tend to lead on the low-precision formats (FP8 and below) and matrix-engine features that AI leans on, while consumer cards prioritize the graphics features games need.
• Form factor, cooling, and price. Datacenter parts are built for dense racks, aggressive cooling, and continuous full-load operation, and are priced for enterprises, not enthusiasts.

The through-line: consumer and datacenter GPUs diverge exactly along the axes that AI cares about — memory bandwidth, memory capacity, and the ability to cooperate. Everything on the datacenter side is a direct answer to a bottleneck this article has already named.

The software moat: CUDA and why the ecosystem is the product

Here's a fact that surprises people: NVIDIA's most durable advantage may not be its silicon at all. It's the software. For roughly two decades the company has built and given away CUDA — the programming platform, compilers, and above all the deep libraries that let developers actually use all that parallel hardware without writing the fiendishly difficult low-level code themselves. Nearly every AI framework is built on top of these libraries. When a researcher writes a few lines of high-level Python and it runs fast on a GPU, an enormous stack of NVIDIA-optimized software is doing the heavy lifting underneath.

This matters for understanding the industry because it explains why raw hardware competition is so much harder than "build a faster chip." A rival can design an accelerator with more FLOPS or more bandwidth on paper and still lose, because the moment a customer tries to run their existing models on it, nothing is optimized, the libraries are immature, the edge cases break, and the promised performance evaporates. The chip is only as good as the software that feeds it work — and a mature software ecosystem is the accumulated labor of tens of thousands of engineer-years that a competitor cannot simply copy. This is a genuine moat, and it's why the industry pours effort into portable software layers and compilers meant to break the lock-in: the hardware race is real, but the software race quietly decides a lot of it. When you evaluate any "GPU killer" claim, the first skeptical question isn't "how fast is the chip" — it's "what does it take to actually run my models on it, and how much performance survives the translation."

How much GPU do you actually need?

The honest answer is another question — for what? — but the frameworks in this article let you reason it out instead of guessing, so here's the decision tree.

First, is it training or inference? Training a model from scratch, or heavily fine-tuning one, is a job for clusters and is out of reach for almost everyone outside well-funded labs; assume the cloud. Most people asking this question actually mean inference — running an existing model — which is far more tractable.

For inference, the first gate is capacity: do the weights even fit in the memory? A model's size in memory is roughly its number of parameters times the bytes per parameter — and the bytes per parameter is set by the precision, which is where quantization becomes the single biggest lever for fitting a model onto modest hardware. Shrink the numbers and a model that wouldn't fit suddenly does. If the weights don't fit, nothing else matters; you're stuck swapping data in and out and performance falls off a cliff.

The second gate is bandwidth: can you stream the weights fast enough to be usable? Two machines that both hold a model can feel completely different to use, because generation speed in the memory-bound decode phase is set by how fast weights move, not by how many multipliers you have. A consumer machine can often hold a smaller model but streams it slowly, which is exactly why local models feel sluggish next to a datacenter — same architecture, narrower pipe. The full, practical walkthrough of picking hardware and model size for your own machine is in the run LLMs locally guide; the point here is that "how much GPU" decomposes cleanly into capacity (a fit/no-fit question answered mostly by model size and precision) and bandwidth (a speed question answered by the memory system). Answer those two and you've answered the question — without needing a single benchmark or product name.

Why this one idea explains the whole industry

Almost every confusing thing about the business of AI dissolves once you hold the hardware picture straight.

Scarcity. Frontier accelerators are hard to make — the exotic memory and advanced fabrication are supply-constrained, and demand is effectively unlimited. When you read about companies hoarding chips or waiting months for delivery, that's the parallelism-and-bandwidth machine being the one bottleneck nobody can route around.

Cost. Running AI is expensive because generation is memory-bound and the chips that do it well are scarce and power-hungry. The price of a token traces directly back to bandwidth and silicon — and so does the energy and water footprint those power-hungry chips run up. This is also why so much engineering effort goes into batching, quantization, and smarter serving — each squeezes more useful work out of the same bottleneck.

Local vs cloud. Whether you can run a model on your own machine comes down to two hardware facts: do the weights fit in your memory, and is your memory bandwidth high enough to stream them at a usable speed? A consumer machine can often hold a smaller model but streams its weights slowly, which is exactly why local models feel sluggish next to a datacenter. Same architecture, smaller pipe.

The pace of progress. A lot of AI's forward motion is hardware motion — more bandwidth, more parallel cores, better interconnects, smaller number formats. Software cleverness matters enormously, but it's climbing a ladder the hardware keeps extending. When you think about where AI goes next, a big part of the answer is written in silicon roadmaps.

You don't need to know a single product name to reason about any of this. You need the two ideas: parallel matrix multiply, and memory bandwidth as the ceiling. Everything else is detail hanging off that frame.

FAQ

Can't a CPU run AI? Why does it specifically need a GPU? A CPU absolutely can run a neural network — it's just doing the same matrix multiplications, only a few at a time instead of thousands. For a small model or a low-traffic task, that's fine. For anything at modern scale it's impractically slow: a CPU might take minutes to do what a GPU does in a fraction of a second, because AI's workload is thousands of independent operations at once and a CPU can only run a handful in parallel. It's not that CPUs can't; it's that they're the wrong shape for this specific job.

What actually makes a GPU faster for AI — is it just doing more math per second? That's the common assumption, and it's only partly true. Yes, a GPU does far more arithmetic per second thanks to thousands of parallel cores. But in practice the arithmetic units are often idle, waiting for data to arrive from memory. For a lot of AI work — especially generating text token by token — the limit is memory bandwidth (how fast numbers move from memory to the cores), not raw compute. The GPU wins on parallel math and on having fast enough memory to feed that math.

What does "memory bandwidth" mean and why does everyone say it's the bottleneck? Memory bandwidth is the rate a chip can move data between its memory and its compute cores. It's the bottleneck because moving a number is far slower and more energy-hungry than multiplying it — a chip can do many multiplies in the time it takes to fetch one value from memory. Generating text means reading a large fraction of a model's billions of weights from memory for every token, with each weight used only once. So how fast the model responds is set mostly by how fast weights stream out of memory, not by how fast the chip can multiply.

Are the chips running AI still really "GPUs"? Loosely. They descend from graphics chips and kept the name, but many have no display output and are built purely for AI: dedicated matrix-multiply cores, high-bandwidth memory stacked next to the compute, and fast links to connect thousands of them. "AI accelerator" is the more accurate term. The underlying idea — massively parallel arithmetic fed by fast memory — is the same one that made GPUs good at graphics, which is why the lineage stuck.

Why are AI chips so expensive and hard to get? Because the things that make them good — huge parallel compute plus exotic high-bandwidth memory plus advanced manufacturing — are all supply-constrained, while demand is effectively unlimited. The high-bandwidth memory in particular is difficult and costly to produce. Since these chips are the one component you genuinely can't route around for large-scale AI, whoever controls the supply controls the pace, and prices reflect that scarcity.

Why does a model need many GPUs instead of one big one? Two reasons. First, capacity: a frontier model's weights can run to hundreds of gigabytes, far more than fits on any single chip, so the model has to be split across many. Second, speed: even when a model fits, spreading the work across chips lets it finish faster. The catch is that split-up chips must constantly exchange data, so the interconnect wiring them together — fast short-range links inside a server, high-speed networking between servers — becomes as important as the chips. The goal is to make a whole rack behave like one enormous GPU, and the networking to pull that off is often harder engineering than the chips.

What are tensor cores, and why does "lower precision" keep coming up? Tensor cores (other vendors call them matrix engines) are hardware blocks that do nothing but small matrix multiplies — the exact operation a neural network is made of — so dedicating silicon to them gives a huge speedup. They pair with lower-precision number formats (16-bit, 8-bit, and narrower) because neural networks tolerate imprecise arithmetic well, and fewer bits per number buys two things at once: more multiplies per second and fewer bytes to move from memory. Since bandwidth is usually the ceiling, shrinking the numbers attacks both bottlenecks simultaneously, which is why every new chip generation reaches for a narrower format.

Are GPUs the only option, or do TPUs and other chips matter? GPUs aren't the only accelerator. Some operators build custom chips (TPUs and other ASICs) that hard-wire the same ideas — parallel matrix multiply, fed by fast memory — even more aggressively, trading general-purpose flexibility for more efficiency on one workload. Conceptually they're the same bet, not a different one. The tradeoff is rigidity: a fixed-function chip can win on efficiency for the exact models it was built for, but a flexible GPU can often just run whatever comes next while specialized silicon may need a years-long redesign. There's no permanent winner, only a moving frontier.

Do I need to understand GPUs to use or build AI products? To use AI, no. To build with it, understanding the shape helps a lot. Knowing that generation is memory-bound explains why batching cuts your costs, why smaller number formats speed things up, why bigger context windows get pricey, and why a local model streams slower than a cloud one. You don't need product names or benchmarks — just the two core ideas. If you're choosing an LLM for an app, this frame tells you where the cost and latency actually come from.

The one-sentence version

A GPU is a machine for doing thousands of independent multiplications at once — which is exactly what a neural network is made of — and the real limit on modern AI isn't how fast those multiplications happen, but how fast the numbers can be fetched from memory to feed them. Hold those two facts and the rest of AI hardware is just details.