How Transformers Actually Work: A Visual Guide to Attention

Almost every AI system you've heard of — the chatbots, the coding assistants, the image describers, the voice models — runs on the same underlying architecture, called a transformer. And the transformer, stripped of its jargon, is built around a single idea: attention. That one mechanism is what let the field leap from "sometimes-coherent autocomplete" to systems that hold a thread across thousands of words. If you understand attention, you understand the load-bearing wall of modern AI.

Here's the whole idea in one sentence: attention lets every word in a sentence look at every other word and decide which ones matter for interpreting it. That's it. The rest of this guide unpacks what "look at" means, why it beat the previous generation of models so decisively, and what the intimidating terms — queries, keys, values, multi-head — actually refer to. We start with pure intuition and no equations, then, for readers who want the machinery, we open the hood and walk through the actual arithmetic, one dot product at a time. You can stop after the intuition and still understand the load-bearing idea; keep reading and you'll understand how a real transformer block is wired.

Table of contents

1. Key takeaways
2. Why the old way hit a wall
3. Self-attention, in plain language
4. Queries, keys, and values
5. Self-attention, step by step: the actual math
6. Why multiple heads
7. Inside one transformer block: LayerNorm, residuals, and the FFN
8. The residual stream: a shared workspace
9. The order problem
10. Positional encodings: from sinusoids to RoPE
11. Stacking it up: from attention to a model
12. Encoder, decoder, encoder-decoder, and causal masking
13. What attention costs
14. How transformers differ from RNNs and CNNs
15. Common misconceptions
16. FAQ
17. The one thing to remember

Key takeaways

• Attention is the breakthrough. Transformers replaced older sequence models (RNNs) because attention lets a model relate any two words directly, no matter how far apart, and process them all at once instead of one at a time.
• Self-attention is words looking at words. Each word gathers context from the other words in the sentence, weighting them by relevance, to build a richer version of itself.
• Queries, keys, and values are a lookup system. Each word broadcasts what it's looking for (query), what it offers (key), and what it actually contributes (value). Matches get more weight.
• Multi-head attention runs several of these lookups in parallel, so the model can track grammar, meaning, and reference relationships at the same time.
• Transformers have no built-in sense of order, so word position has to be added in explicitly. This is a feature — it's why they parallelize so well — not an oversight.
• Attention is powerful but not free: its cost grows with the square of the input length, which is the root of most "context window" limitations.
• Attention is not the whole transformer. Each layer pairs attention with a small per-word neural network (the feed-forward layer), wrapped in normalization and residual connections. Attention moves information between positions; the feed-forward layer does the thinking at each position. You need both.

Why the old way hit a wall

Before transformers, the dominant approach to language was the recurrent neural network (RNN), and its more capable cousin the LSTM. The mental model is a reader with a notebook. The model reads one word, updates a running summary in the notebook, reads the next word, updates again, and so on to the end of the sentence. Everything it knows about the earlier text is squeezed into that single running summary.

This has two crippling problems.

First, information decays over distance. By the time the model reaches the end of a long paragraph, the opening words have been overwritten and diluted many times. If a pronoun at the end refers back to a name at the start, the connection is often already lost. People tried to patch this — LSTMs added gates to protect important memories — but the fundamental bottleneck remained: everything had to pass through one narrow summary that got rewritten at every step.

Second, and just as important for the economics, RNNs are sequential by construction. You cannot read word five until you've finished processing word four, because word four's output feeds into word five. That makes them painfully slow to train, because you can't take advantage of hardware that's built to do thousands of things simultaneously. Modern AI accelerators are massively parallel; RNNs leave most of that parallelism on the table.

There's a subtler third problem that gets less attention but matters enormously in practice: the path length between two related words scales with their distance. For an RNN to connect word 1 to word 50, information has to survive 49 sequential updates, each one multiplying and re-mixing the hidden state. This is the mechanical root of the "vanishing gradient" problem — during training, the learning signal that should teach the model "word 50 depends on word 1" has to propagate back through all 49 steps, and it shrinks (or explodes) exponentially along the way. LSTMs and their gates were, in essence, an elaborate scheme to build a protected highway through those steps so some gradient could survive. It helped, but it never removed the underlying fact that the path was long.

Attention fixes all of this at once. It lets any word reach any other word in a single step — the path length between any two positions is constant, regardless of how far apart they sit — with no relay race and no lossy summary. And because no word's computation depends on having finished the previous word's, the whole sentence is processed in parallel. That combination — constant path length plus full parallelism — is why the 2017 paper introducing the transformer was, fittingly, titled "Attention Is All You Need." The provocative claim in that title was that you could throw away recurrence entirely, keep only the attention mechanism (which had previously been a helper bolted onto RNNs), and end up with something strictly better. It was right.

Self-attention, in plain language

Take the sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to — the animal or the street? You know instantly it's the animal, because "tired" applies to animals, not streets. To resolve "it," you didn't reread the sentence from the start holding everything in memory. You just let "it" look at the other words and latch onto the one that mattered.

That is self-attention. For every word in the input, the model asks: given what I am, which other words should I pay attention to in order to understand myself in this context? The word "it" pays a lot of attention to "animal," a little to "tired," and almost none to "street." The result is a new, context-enriched representation of "it" that effectively carries the meaning "it (= the animal)."

Crucially this happens for every word at the same time, and every word gets to attend to every other word — including itself. "Cross" attends to "street" and "animal." "Tired" attends to "animal" and "it." The model builds up a web of relationships across the whole sentence in a single pass. That web is the context, and it's far richer than any single running summary could ever be.

One point that trips people up: the words don't start as bare symbols. Before any attention happens, each word (technically, each token — a word or word-fragment) is turned into a list of numbers called an embedding, a point in a high-dimensional space where geometrically nearby points mean semantically similar things. Attention operates on these vectors, not on letters. So when we say "it" absorbs the value of "animal," what physically happens is that the numeric vector representing "it" gets nudged in the direction of the vector representing "animal." After the attention step, the vector at the "it" position is literally a different set of numbers than it was before — one that now encodes "the animal, which was too tired." That updated vector is what the next layer sees. Self-attention, then, is a machine for rewriting each word's vector using context drawn from the other words' vectors.

Notice also what self-attention is not. It is not a search that returns one winner. Every word contributes something to every other word; it's just that most contributions are tiny. The output for "it" is a blend that's, say, 70% "animal," 12% "tired," and thin slivers of everything else. This soft, everything-at-once weighting is exactly what makes attention differentiable and therefore trainable — there's no hard "pick the best match" decision that gradients can't flow through.

Queries, keys, and values

So how does a word "decide" what to pay attention to? This is where queries, keys, and values come in — the three terms that scare people off. They're actually a familiar idea: a search or lookup system, like a dating app or a library catalog.

Every word produces three things:

• A query: what am I looking for? ("it" is looking for the noun it refers to.)
• A key: what do I offer, how should others find me? ("animal" advertises itself as a concrete noun that can be an antecedent.)
• A value: what information do I actually pass along if I'm selected? ("animal" hands over its meaning.)

The matching works like this. Each word's query is compared against every word's key. A strong match produces a high score; a weak match, a low one. Those scores are turned into weights that add up to 100%. Then each word builds its updated representation by taking a weighted blend of all the values — mostly the values of the words it matched strongly, a trace of the rest.

Think of "it" walking down the sentence holding up a sign that says "I need a noun that can be tired." Every other word holds up its own key-sign. "Animal" is the best match, so "it" absorbs most of "animal's" value. That's a single attention step.

Term	Everyday analogy	What it answers
Query	Your search box text	"What am I looking for?"
Key	A page's title/tags	"What am I, so others can find me?"
Value	The page's actual content	"What do I contribute if chosen?"
Attention weight	Search relevance score	"How much should I care about this word?"

One subtlety worth naming: query, key, and value are all derived from the same word, but through three different learned transformations. The model learns how to turn a word into a good query, a good key, and a good value during training — nobody hand-codes "animals can be antecedents." It falls out of the model adjusting itself over billions of examples. (If the query/key matching feels reminiscent of how vector search and embeddings find similar items by comparing directions in space, that's not a coincidence — it's the same underlying idea of similarity by dot product.)

Those "three different learned transformations" are three matrices, usually written W_Q, W_K, and W_V. Each is a rectangular grid of numbers the model tunes during training. To get a word's query, you multiply its embedding vector by W_Q; for the key, by W_K; for the value, by W_V. These three matrices are shared across every position in the sentence — the word "animal" and the word "street" both pass through the identical W_Q — which is exactly what lets the mechanism generalize to sequences of any length and to inputs it has never seen. What differs between words is only their embeddings going in; the transformation applied is the same.

Why bother separating key from value at all? Because how findable you are and what you contribute are genuinely different jobs. A word might be a strong match for a query (high key alignment) while the information it should hand over — its value — lives in a different subspace entirely. Splitting the two lets the model decouple "who should attend to me" from "what they receive when they do." Collapse key and value into one vector and you force those roles to share the same numbers; the model loses a degree of freedom it clearly finds useful.

Self-attention, step by step: the actual math

Everything above was intuition. Here is the machinery, and it's less scary than its reputation. The entire operation fits in one line that you'll see written as:

Attention(Q, K, V) = softmax( QKᵀ / √d ) V

Let's take it apart piece by piece, because each symbol corresponds exactly to something we already described in words.

Step 1 — Build Q, K, V. Stack every word's embedding into a matrix (one row per word). Multiply that matrix by W_Q, W_K, and W_V to get three matrices: Q (all the queries), K (all the keys), V (all the values). One matrix multiply produces the queries for the entire sentence at once — this is the parallelism, made concrete.

Step 2 — Score every pair (QKᵀ). The expression QKᵀ compares each query against each key by dot product. The dot product of two vectors is large and positive when they point in similar directions, near zero when they're unrelated, negative when they point apart. So QKᵀ produces a square grid of scores: for a 10-word sentence, a 10×10 table where cell (i, j) says "how much does word i's query match word j's key?" This grid is the "web of relationships" from earlier, in numeric form. Row "it" will have a big number in the "animal" column.

Step 3 — Scale by √d. Notice the division by √d, where d is the dimension of the key vectors (the length of each key list). This is the one part people skip, and it matters. When d is large — say 64 or 128 — dot products tend to grow large in magnitude simply because you're summing over many terms. Feed very large numbers into the next step (softmax) and it saturates: one weight rounds to ~1.0 and the rest to ~0.0, attention becomes a hard, brittle pick, and the gradients that should train the model vanish. Dividing by √d keeps the scores in a sane range so learning stays stable. It's a small factor with an outsized effect on whether training works at all.

Step 4 — Softmax into weights. Softmax takes each row of scores and squashes it into positive numbers that sum to 1 — a probability distribution across the words. Bigger scores get exponentially more weight (softmax exponentiates before normalizing), which is why a clearly-best match dominates while still leaving crumbs for the runners-up. After this step, row "it" might read something like 0.70 on "animal," 0.12 on "tired," and the remaining 0.18 spread thinly across everything else. These are the attention weights.

Step 5 — Blend the values (× V). Multiply the weight grid by V. Each word's output becomes the weighted sum of all value vectors, using that word's row of weights. "It" walks away with 70% of animal's value vector added to 12% of tired's, and so on — a brand-new vector that encodes "it, meaning the animal." Repeat for every row, and you've updated every word in the sentence simultaneously.

That's the whole thing. Five steps, one of which (the scaling) is a numerical safety measure. A worked micro-example makes the shape obvious: with 4 words and, say, key dimension 3, Q is 4×3, K is 4×3, so QKᵀ is 4×4 (every word against every word), softmax leaves it 4×4, and multiplying by V (4×3) returns a 4×3 result — same shape as the input, one updated vector per word. Attention is a shape-preserving operation: words in, the same number of words out, each one rewritten with context. That's precisely why you can stack it over and over.

Why multiple heads

There's a problem with a single attention mechanism: a word usually needs to track several kinds of relationships at once, and one weighted blend can only express one kind of "relevance" at a time.

Consider the word "cross" in our sentence. Grammatically, it wants to find its subject ("animal") and its object ("street"). Semantically, it might want to relate to "street" as a location. Those are different questions, and forcing them through a single query/key comparison muddies both.

Multi-head attention solves this by running several independent attention operations in parallel — typically anywhere from a handful to dozens — each with its own separate queries, keys, and values. Each "head" learns to specialize. When researchers inspect trained models, they often find heads that have quietly picked up distinct jobs: one tracks subject–verb agreement, another links pronouns to their antecedents, another follows positional patterns like "the word right before me." No one assigned these roles; the heads differentiated on their own because it helped the model predict better.

The intuition: instead of one person reading the sentence and forming one impression, you have a committee. One member watches grammar, one watches meaning, one watches word order. Afterward their notes are combined into a single richer understanding of each word. More heads means more relationships trackable in parallel — at the cost of more computation.

Mechanically, multi-head attention doesn't just run the full-size attention operation several times. It splits the vector into slices. If a model works with 512-dimensional vectors and uses 8 heads, each head gets its own 64-dimensional slice, with its own small W_Q, W_K, W_V operating only on that slice. Each head runs the five-step attention procedure inside its 64-dimensional world, producing a 64-dimensional output. The eight outputs are then concatenated back into a 512-dimensional vector and passed through one more learned matrix (often called the output projection, W_O) that mixes the heads' findings into a single coherent update. This is a deliberate design choice: it means multi-head attention costs roughly the same as single-head attention of the same total width, because the work is divided among heads rather than multiplied by them. You get several specialized views for the price of one wide one.

A useful nuance for anyone reading about model internals: interpreting a single head's job is tempting but risky. Researchers have found heads that clearly do one thing — a "previous token" head, an "induction" head that completes repeated patterns — but many heads are polysemantic, doing different jobs in different contexts, and some appear nearly redundant (you can prune them with little loss). Treat "this head handles pronouns" as a helpful story, not a hard fact. The specialization is real in aggregate; it's just messier than the tidy committee metaphor suggests. There's also a memory consequence to head count that shows up at inference time — the keys and values of past tokens have to be cached, and that KV cache grows with the number of heads, which is why modern models often share keys and values across groups of heads (grouped-query attention) to shrink it.

Inside one transformer block: LayerNorm, residuals, and the FFN

Attention gets all the headlines, but a transformer "block" — the unit that gets stacked dozens of times — has more moving parts, and each one is there for a reason. A single block does roughly this, in order: normalize, attend, add back, normalize, feed-forward, add back. Two of those words — add back and normalize — are as important as the attention itself.

The feed-forward network (FFN) is where much of the actual computation lives. After attention has mixed information across positions, each word's vector is passed, independently, through a small two-layer neural network: expand it to a larger width (often 4× the model dimension), apply a nonlinearity, then project back down. This happens per position, in parallel, with no interaction between words. If attention is the step where words talk to each other, the FFN is the step where each word sits alone and thinks about what it just heard. It's also where a large fraction of a model's parameters — and a large fraction of its stored factual knowledge — actually reside. A common surprise for newcomers: in most large models, the feed-forward layers hold more weights than the attention layers do. Attention is the clever part; the FFN is the heavy part.

Residual connections (the "add back") are what let you stack deeply at all. Instead of replacing a word's vector with the attention output, the block adds the attention output to the original vector. Same for the FFN. Written plainly: output = input + attention(input), then output = output + ffn(output). This looks like a triviality; it isn't. It means each sublayer only has to learn a change — a nudge to the running representation — rather than reconstruct the whole thing from scratch. It also gives gradients a clean, near-unobstructed path from the top of a 96-layer stack back down to the bottom during training. Without residual connections, deep transformers simply don't train; the signal degrades through the layers exactly the way it did in deep networks before this trick became standard.

LayerNorm (the "normalize") keeps the numbers well-behaved. Before (or after, depending on the design) each sublayer, the vector is normalized — rescaled so its values have a consistent statistical spread — then given two small learned knobs to adjust that scale and shift. The purpose is stability: as vectors get added to over and over by residual connections, their magnitudes can drift and blow up. Normalization reins them back in at each step so the next sublayer always receives inputs in a predictable range. Modern models mostly apply it before each sublayer ("pre-norm"), which trains more reliably than the original "post-norm" placement — one of those small architectural refinements that quietly made very deep transformers routine.

Put together, one block is: normalize → multi-head attention → add back → normalize → feed-forward → add back. Stack that structure N times, and you have the body of a transformer. Everything else — embeddings at the bottom, a prediction layer at the top — bookends this repeating sandwich.

The residual stream: a shared workspace

Here's a mental model that makes deep transformers click, and it follows directly from those residual connections. Picture a single vector per word flowing straight up through the entire stack, from the first layer to the last, like a conveyor belt. Every attention sublayer and every feed-forward sublayer reads from this belt, computes something, and writes its result back by adding it on. Nothing is ever overwritten; contributions only accumulate. This running vector is called the residual stream, and it's arguably the most useful abstraction for reasoning about what transformers do.

Under this view, the layers don't pass a baton so much as they all scribble onto a shared notepad. An early attention head might write "this token is a subject noun" into the stream. A middle-layer head reads that annotation and writes "its verb is three words ahead." A feed-forward layer reads the accumulated context and writes in a factual association. Because it's all addition into a common space, later layers can pick up what earlier layers deposited — and the model learns to route information through this shared channel.

This reframes the "stacking" question. Depth isn't just repetition; it's iterative refinement of a single evolving representation. Each word's vector at the top of the stack is the sum of its starting embedding plus every nudge every layer decided to contribute along the way. When researchers study models by "reading off" the residual stream at intermediate layers, they can watch a prediction take shape gradually — a fuzzy guess early on sharpening into a confident answer near the top. The residual stream is also why techniques like adding a steering vector, or inspecting where a fact is "stored," even make sense: there's a single, additive, interpretable channel to intervene on.

The order problem

Here's a consequence of processing every word simultaneously that surprises people: a raw transformer has no idea what order the words are in. To the attention mechanism, a sentence is more like a bag of words all shouting at each other at once. "Dog bites man" and "man bites dog" would look identical, which is obviously catastrophic for language.

The fix is to inject position information directly into each word before attention happens. Every word gets tagged with a signal that encodes where it sits in the sequence — first, second, seventeenth. There are several schemes for doing this (early transformers added fixed wave-like patterns; most current models use rotation-based methods applied inside the attention step), and the details are one of the more active areas of architecture research, especially for stretching models to longer inputs. But the concept is durable: because transformers threw away sequential processing to gain parallelism, order has to be handed back to them explicitly.

This is the trade at the heart of the architecture. RNNs got word order for free but paid with a sequential bottleneck. Transformers gave up the free order to unlock parallelism and long-range connections — then bought order back cheaply with position tags. It turned out to be one of the best trades in the history of the field.

Positional encodings: from sinusoids to RoPE

The word "tag" glosses over some genuinely clever engineering, and since position handling is where a lot of the "why does my model get worse on long inputs?" behavior comes from, it's worth a closer look.

The original approach: sinusoidal encodings. The 2017 transformer added a fixed pattern to each word's embedding — a set of sine and cosine waves of different frequencies, one combination per position. Position 1 got one signature of wave values, position 2 a slightly different one, and so on. Different dimensions of the vector oscillated at different rates, so the pattern was unique per position but also smooth: nearby positions got similar signatures. The elegant property was that the difference between position 5 and position 8 looked consistent regardless of where in the sentence you were, giving the model a way to reason about relative distance. Because it was a fixed formula rather than learned, it could in principle extend to positions longer than anything seen in training — though in practice models trained this way still degrade when pushed well past their training length.

A simpler variant: learned absolute positions. Many models just gave each position its own learnable embedding vector, added to the word embedding — position 1 has a vector, position 2 has a vector, up to some maximum. Simple and effective, but with a hard ceiling: a model that only ever learned position vectors up to 2,048 has literally no representation for position 2,049. This is one concrete reason context windows used to be fixed hard limits.

The modern default: rotary position embeddings (RoPE). Instead of adding a position signal to the embedding up front, RoPE rotates the query and key vectors by an angle proportional to their position, applied inside the attention step. The beautiful consequence: when a query at position i and a key at position j meet in a dot product, the math works out so that what survives depends on their relative offset (i − j), not their absolute positions. Attention naturally becomes a function of "how far apart are these two words," which is usually what you actually want. RoPE is why so much recent work on extending context length focuses on manipulating those rotation frequencies — techniques with names like frequency scaling or "NTK-aware" interpolation are all tricks to make rotations that were learned at short lengths behave sensibly at longer ones, so a model trained on modest inputs can be stretched to larger windows without full retraining. If you want the deep version of why long inputs strain attention specifically, that's the subject of long-context attention.

The throughline across all three schemes: position is not fundamental to attention, it's an additive accessory. That's a strength — it's tweakable, swappable, and extendable independent of the rest of the model — and a weakness, because a model's grasp of position is only as good as the scheme bolted on, which is exactly why "lost in the middle" behavior and long-context degradation so often trace back to how positions were encoded.

Stacking it up: from attention to a model

One attention step gives each word a context-aware update. Real models stack these — dozens of layers, each with its own multi-head attention followed by a small standard neural network that processes each word individually. The output of one layer feeds the next.

Why stack? Because meaning is hierarchical. Early layers tend to capture local, surface patterns — nearby words, simple grammar. Middle and later layers compose those into more abstract relationships — who did what to whom, tone, longer-range references, task intent. It's loosely analogous to how vision models build from edges to shapes to objects. By the top of the stack, each word's representation is soaked in context from the entire input, filtered through many rounds of "which other words matter to me?"

That deep, context-saturated representation is what actually drives the model's behavior. In a chatbot, the model uses the representation of the final position to predict the next word, then repeats — the loop we cover in how AI chatbots actually work. Attention is the engine; next-word prediction is the steering wheel bolted on top.

Worth flagging a subtlety about "early layers do grammar, late layers do meaning": it's a genuine tendency, not a law. The specialization emerges from training, isn't cleanly separable, and varies between models. Treat it as a reliable direction of travel — representations get more abstract and more global as you go up — rather than a labeled floor plan. What is robust is that a token's final-layer vector has, through the residual stream, been influenced by essentially the entire input, refracted through dozens of rounds of relevance-weighting.

Encoder, decoder, encoder-decoder, and causal masking

Not all transformers are wired the same way, and the differences explain why some models are built for understanding text and others for generating it. The distinction comes down to one question: when a word attends, is it allowed to look at words that come after it?

Encoders let every word see every other word — past and future. This is bidirectional attention. When the goal is to understand a fixed piece of text (classify its sentiment, find the entities in it, produce an embedding of the whole thing), there's no reason to hide the future — the whole input is available at once, so let each word draw on full context in both directions. Encoder-style models excel at analysis tasks and at producing rich representations of existing text. Their limitation is that they don't naturally generate; they're readers, not writers.

Decoders can only look backward. This is where causal masking comes in, and it's a beautifully simple trick. Recall that attention produces that grid of scores, one row per word, one column per word. To prevent a word from attending to future words, you take the upper triangle of that grid — every cell where a word would be looking ahead — and set those scores to negative infinity before the softmax. Softmax turns negative infinity into a weight of zero. The effect: word 5 can attend to words 1 through 5, but words 6, 7, 8 are invisible to it, as if they didn't exist. This is what makes a decoder a valid next-word predictor. If a word could peek at the future during training, predicting that future would be trivial cheating — the model would learn nothing useful. The mask enforces the honest rule: predict what comes next using only what came before. Every mainstream generative LLM is a decoder-only transformer built on this masked, backward-looking attention.

Encoder-decoder models use both, joined by cross-attention. The original transformer was designed for translation, which naturally splits into two jobs: read the whole source sentence (encoder, bidirectional), then generate the translation one word at a time (decoder, causal). The bridge between them is cross-attention: in the decoder, some attention layers form their queries from the sentence being generated but draw their keys and values from the encoder's output. In plain terms, the sentence being written gets to look at the source sentence at every step. This design still dominates tasks with a clear input-to-output mapping — translation, summarization, speech-to-text — where you want to fully digest an input before producing an output.

The key realization is that these are three configurations of the same parts, not three different architectures. Same attention, same feed-forward, same residual stream. What changes is the masking (can you see the future?) and whether there's a second stack feeding in through cross-attention. The industry's convergence on decoder-only models for general-purpose LLMs is largely because the "predict the next token" objective is simple, scales beautifully, and turns out to teach the model to both understand and generate at once.

What attention costs

Attention's superpower — every word can look at every other word — is also its main expense. If a sentence has 100 words, each word compares against 100 keys, so you're doing on the order of 100 × 100 = 10,000 comparisons. Double the length to 200 words and it's 40,000 — four times the work for twice the text. This quadratic growth is why doubling an input can more than double the compute, and it's the root cause behind context-window limits, the higher price of long prompts, and a whole research industry devoted to cheaper approximate attention.

It also explains a practical asymmetry worth internalizing: a longer prompt doesn't just cost proportionally more, it costs disproportionately more inside the attention step. If you care about the dollars-and-cents side of this, we go deep in AI inference cost economics — but the architectural reason traces straight back to that all-pairs comparison. When you hear about "sparse attention," "sliding windows," or "linear attention," they're all attempts to avoid comparing literally every pair while keeping most of the benefit.

Two refinements make this picture more accurate. First, the quadratic term isn't the only cost — the feed-forward layers scale linearly with length and, for short-to-moderate inputs, often dominate the actual runtime. Attention's quadratic term only takes over and becomes the bottleneck once sequences get long, which is precisely when it hurts. Second, the more binding constraint at inference time is frequently memory, not arithmetic. Every token you've already processed leaves behind its keys and values, cached so they don't have to be recomputed each step. That cache grows linearly with sequence length and can consume more memory than the model's own weights on long conversations — which is why so much systems work targets shrinking it (grouped-query attention, quantized caches, paged memory). The all-pairs comparison sets the compute bill; the KV cache sets the memory bill; both scale with length, which is why length is the single most important number for the economics of running a model.

There's a genuine trade in the "cheaper attention" schemes worth naming honestly: none is free. Sliding-window attention (each word only sees the last k words) makes cost linear but severs true long-range links, leaning on stacked layers to relay information indirectly. Linear-attention variants approximate the softmax and often lose a little of the sharp, selective focus that made attention work in the first place. The full quadratic version remains the quality benchmark; the approximations are bets that you can drop most of the pairwise comparisons and barely notice — bets that pay off for some tasks and quietly degrade others.

How transformers differ from RNNs and CNNs

It's clarifying to place the transformer next to the two architectures it displaced, because each makes a different bet about how information should move through a sequence.

RNNs move information step by step. To relate word 1 and word 50, an RNN threads the connection through 49 intermediate states. Path length is long, processing is inherently sequential, and memory is a single fixed-size hidden state that everything must squeeze through. The upside — rarely worth it for language — is that an RNN's cost per token is constant regardless of history length, and it has an unbounded (if lossy) notion of the past. Transformers traded that constant-memory property away for direct access and parallelism.

CNNs (convolutional networks) move information through local windows. A convolution looks at a small neighborhood — a few adjacent words — and detects patterns within it. To relate distant words, you stack convolutions until their receptive fields overlap; connecting word 1 to word 50 might take many layers of widening windows. CNNs parallelize well (unlike RNNs) but reach long distances only through depth, and their sense of "what to look at" is fixed by the window shape rather than chosen dynamically from content.

Transformers move information by content-based, all-pairs routing. Any word can reach any other in one step, and which words connect is decided on the fly by the query-key match rather than fixed by position or window. That's the essential difference: RNNs and CNNs have hard-wired information-flow patterns (sequential chain, local window); attention learns its wiring from the data, per input, every layer. The cost of that flexibility is the quadratic all-pairs comparison. The benefit is that a model can, in a single layer, decide that the relevant context for this word is another word forty tokens away — something neither predecessor can do without many layers of indirection. That single capability, more than raw size, is what unlocked modern language models.

Common misconceptions

A few beliefs about transformers are common, intuitive, and wrong. Clearing them up sharpens the whole picture.

"Attention is a search that picks the most relevant word." It's a soft, weighted blend of all words, not a pick. Every position contributes to every other; the weights just concentrate on the relevant ones. This softness is not a detail — it's what makes the whole thing trainable, because gradients can flow through a smooth blend but not through a hard selection.

"The transformer is just the attention mechanism." Attention is one of two workhorses. The feed-forward layers hold most of the parameters and much of the stored knowledge, and the residual connections plus normalization are what make deep stacks trainable at all. A transformer with attention but no FFN would be nearly useless.

"More attention heads always means a smarter model." Heads split a fixed budget rather than adding one. Beyond a point, extra heads slice the representation too thin to be useful, and studies routinely find heads that can be pruned with little effect. Head count is a balance, not a dial that only goes up.

"Transformers understand word order like we do." They have no innate sense of order; it's injected by positional encodings, and their grasp of position is exactly as good as that added scheme. This is why long-context and "lost in the middle" failures so often trace back to how positions were encoded, not to some deeper reasoning limit.

"Each attention head does one clean, human-nameable job." Some do; many don't. Heads are frequently polysemantic, context-dependent, or redundant. "This head tracks pronouns" is a useful story for the tidy cases, not a general law.

"Bigger context window means the model reads the whole thing equally well." A longer window means the model can attend across the whole input, not that it attends evenly. Relevance still concentrates, position encoding still shapes reach, and information in the middle of a long input is often used less effectively than information at the ends.

FAQ

Is a transformer the same thing as a large language model? Not quite. A transformer is the architecture — the design of attention layers stacked together. A large language model is a transformer that's been trained at massive scale on text to predict the next token. Transformers also power image, audio, and multimodal models. So every mainstream LLM is a transformer, but not every transformer is a language model.

What does the "attention" in attention actually mean? It means selectively weighting information by relevance. When a word is processed, attention lets it pull in more information from the words that matter to its meaning and less from the words that don't. The name is deliberately borrowed from the human sense: focusing on what's relevant and letting the rest fade into the background.

Why did transformers beat RNNs so decisively? Two reasons, both structural. RNNs process words one at a time and compress all earlier context into a single running summary, so distant information decays and training can't be parallelized. Transformers let any word relate to any other word directly through attention, and they process the whole sequence at once — which suits modern parallel hardware and preserves long-range connections.

Do I need to understand the math to use AI models well? No. You can be an expert prompter and builder without ever writing an attention equation. But the intuition pays off: knowing that the model relates words by relevance explains why clear, well-structured prompts work better, and knowing attention is quadratic explains why long inputs cost more and why models sometimes lose the thread. For the practical side, see how to write better prompts.

What is multi-head attention in one sentence? It's running several independent attention operations in parallel so the model can track different kinds of relationships — grammar, meaning, word order, references — simultaneously, then combining their findings into one richer representation per word.

Why do transformers need positional encoding? Because attention treats the input as an unordered set — every word looks at every other word with no built-in notion of sequence. Since word order carries meaning ("dog bites man" ≠ "man bites dog"), position has to be added explicitly by tagging each word with a signal that encodes where it sits. Modern models mostly do this by rotating query and key vectors (RoPE) so attention depends on the relative distance between words rather than their absolute positions.

What does the "√d" in the attention formula actually do? It rescales the query-key scores before softmax. Dot products over high-dimensional vectors tend to grow large in magnitude, and large scores push softmax into a saturated, all-or-nothing regime where one weight is ~1 and the rest are ~0 — which makes attention brittle and starves the training signal. Dividing by the square root of the key dimension keeps scores in a stable range so the model can learn. It's a numerical-stability fix, not a modeling choice, but training tends not to work without it.

What's the difference between an encoder and a decoder transformer? An encoder uses bidirectional attention — every word can see every other word, past and future — which suits understanding a fixed input (classification, embeddings, analysis). A decoder uses causal (masked) attention — each word can only see the words before it — which is what makes it a valid next-word predictor and thus a text generator. Every mainstream generative LLM is decoder-only; the masking is enforced by setting future-word scores to negative infinity before the softmax, which zeroes them out.

Is attention the only thing doing work in a transformer? No, and this is a common oversimplification. Each layer pairs attention with a feed-forward network that processes each word independently, and in most models the feed-forward layers hold more parameters and much of the stored factual knowledge. Attention moves information between positions; the feed-forward layer transforms it at each position. Residual connections and normalization then hold the deep stack together. Attention is the distinctive idea, but it's roughly half the machinery.

The one thing to remember

If you forget everything else, keep this: attention is a mechanism for every element to look at every other element and weight them by relevance, all at once. That single move — replacing a lossy, sequential summary with direct, parallel, relevance-weighted access — is the difference between the AI of the 2010s and the AI you use today. The model names will keep changing. The heads will multiply, the position schemes will get cleverer, the context windows will stretch. But the load-bearing idea underneath has been remarkably stable, and it's the one worth carrying with you. For a broader map of the ideas that matter, the AI canon is a good next stop.