KV Cache: The Complete Guide

The KV cache is the single largest piece of state in LLM serving — bigger than the activations, sometimes bigger than the model itself, and the thing that decides how many concurrent users your GPU can hold. Get its math right and capacity planning becomes arithmetic. Get it wrong and you either overpay for HBM or OOM at the worst moment. This guide is the end-to-end reference: per-architecture KV size derivations (MHA, MQA, GQA, MLA), every quantization variant (FP8, INT8, INT4, KIVI, H2O), paged attention, prefix caching, multi-GPU sharding, CPU/NVMe offload, the SSM and hybrid-attention cousins, and the production cost economics. Companion reading: LLM serving, quantization tradeoffs, long-context attention, disaggregated inference. New to the basics? The cache grows with every token, so start with what a context window is and how tokenization works.

~26,000 words. Use the table of contents to navigate.

Table of contents

1. Key takeaways
2. Mental model: KV cache in one minute
3. A short history of KV cache management
4. What is the KV cache?
5. The math: deriving the per-token KV size
6. Per-model worked examples
7. Attention architecture: MHA, MQA, GQA, MLA
   • Multi-Head Attention (MHA)
   • Multi-Query Attention (MQA)
   • Grouped-Query Attention (GQA)
   • Multi-Head Latent Attention (MLA)
   • Sliding-Window Attention (SWA)
   • Sparse attention
   • Linear attention and SSMs
8. Quantizing the KV cache
   • The standard menu of formats
   • FP8 e4m3 vs e5m2
   • Calibration: per-tensor, per-channel, per-token scales
   • When INT4 KV breaks
   • FP4 on Blackwell
   • Enabling KV quantization in practice
9. PagedAttention: the OS-style memory manager
   • The problem before paging
   • The paging insight
   • Concrete utilization numbers
   • Block size: the one knob worth tuning
   • Implementation cost: paged-aware kernels
   • Paged KV in serving stack timelines
   • How paged attention kernels actually work
10. Prefix caching and RadixAttention
11. Multi-GPU: tensor parallelism and KV sharding
    • Tensor parallelism (TP) and KV head sharding
    • Pipeline parallelism (PP) and KV by layer
    • Expert parallelism (EP) for MoE
    • Combined parallelism
    • NCCL communication and the cost of TP
    • Async compute/communication overlap
    • Sequence and context parallelism (SP / CP, ring attention)
    • When to add a GPU vs add a replica
    • NUMA and PCIe topology gotchas
12. Offloading: CPU, NVMe, hierarchical KV
13. Eviction strategies when the cache fills
14. KV cache and speculative decoding
15. Long-context attention: SWA, sparse, linear, SSM, hybrid
    • Sliding-window attention
    • Sparse attention (Longformer, BigBird, NSA)
    • Linear attention and SSMs
    • Mamba and Mamba-2: how the state actually works
    • Hybrid architectures (Jamba layer pattern in detail)
    • Deploying hybrids in 2026
16. Capacity planning: three worked examples
17. Cost economics: why position matters
18. Stack comparison: vLLM, SGLang, TRT-LLM, TGI, LMDeploy, llama.cpp
19. Comparative benchmarks
20. Migration guide
21. Production observability
22. Failure modes and troubleshooting
23. Frequently asked questions — 30 questions
24. Glossary
25. References

---

Key takeaways

The KV cache is the variable memory bill of LLM inference. Per-token size is governed by one formula:

kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

For Llama-3 70B in BF16 with GQA-8: 320 KB per token. At 32k context: 10.5 GB per request. At 128k context: 42 GB per request — already exceeding an H100 80 GB before the model is loaded.

The four levers that determine whether your serving cluster is profitable:

1. Architecture (GQA / MLA) — picked at training time. Modern open-weight models (2024+) have GQA-8; DeepSeek-V2/V3 have MLA which is even more aggressive. If you're on a pre-GQA model, the answer is "use a different model."
2. KV quantization (FP8 / INT4) — pick at deploy time. FP8 is essentially free quality-wise and halves memory; INT4 is workload-dependent.
3. Paging + prefix caching — free if you're on a modern stack. Lift utilization from 30–50% (naive) to 90%+ (paged) and from 0 (no sharing) to 95% hit rate (well-tuned RadixAttention).
4. Eviction policy — only matters at saturation. Recompute is the right default; swap if you have abundant PCIe and frequent preemption.

The honest summary: understand the KV cache and you understand 80% of why one inference deployment is profitable and another is not. The rest of this guide is everything that depends on or extends that one number.

---

Mental model: KV cache in one minute

If you've never thought about KV caching before, the rest of this guide is going to feel like calculus before algebra. Read this section first. The deep math starts in the next one.

The problem has a name: compute overlap. When a transformer generates text autoregressively, every new token has to look at every prior token in the sequence. Without a cache, generating token N means recomputing the keys and values for tokens 1 through N-1 — work the model already did on the previous step. That overlap is quadratic: doubling the context quadruples the work. KV caching is the technique that eliminates it.

The fix is memoization. After computing K and V for a token, store them. When the next token arrives, compute its K and V, append to the store, and read everything you already had. Decode per token becomes linear in context length instead of quadratic. The cache that holds these tensors — across every layer, every head, every position — is the KV cache.

The cache grows by one position per generated token:

Token 1:  compute (K1, V1)        →  cache: [K1] [V1]
Token 2:  compute (K2, V2)        →  cache: [K1, K2] [V1, V2]
Token 3:  compute (K3, V3)        →  cache: [K1, K2, K3] [V1, V2, V3]
…
Token N:  compute (KN, VN)        →  cache: [K1…KN] [V1…VN]

The cache is per-request, lives on the GPU (because attention reads it on every step), and never shrinks during a request. When the request finishes, the cache is freed.

Without cache vs with cache — side-by-side:

Aspect	Without KV cache	With KV cache
Work per generated token	Recompute K, V for all prior tokens	Compute K, V only for the new token
Total cost over N tokens	O(N²) per layer	O(N) per layer
Memory cost	Negligible (no cache)	Linear in context × cache dtype
Latency at long context	Grows steeply	Stays bounded by HBM bandwidth
Practical context limit	A few hundred tokens	Tens to hundreds of thousands

Pseudocode — what a KV cache actually is in code:

class KVCache:
    def __init__(self):
        self.k = None  # shape grows on the sequence dim each step
        self.v = None

    def append(self, k_new, v_new):
        # k_new, v_new: shape [batch, heads, 1, head_dim] for one new token
        if self.k is None:
            self.k, self.v = k_new, v_new
        else:
            self.k = torch.cat([self.k, k_new], dim=2)
            self.v = torch.cat([self.v, v_new], dim=2)
        return self.k, self.v

Real production stacks (vLLM, SGLang, TRT-LLM) don't torch.cat — they preallocate paged blocks and write into them in place, because cat reallocates and copies. But the conceptual operation is the same. In HuggingFace transformers, the entire thing is one flag:

output = model.generate(input_ids, max_new_tokens=300, use_cache=True)  # default

The speedup is large and easy to measure. Generating 300 tokens with a 1.7B model: ~12 seconds with caching, ~60 seconds without. That's ~5× on a small model; on a 70B model with 32k context, the without-cache version is so slow it's effectively unusable (minutes per token).

Why this guide exists. The conceptual story above is one paragraph. The production story is everything that follows: how big the cache actually gets (math section), how to make it smaller (architecture: GQA, MLA; quantization: FP8, INT4), how to share it across users (prefix caching), how to page it (PagedAttention), what happens when it overflows (eviction), and what it costs in dollars (economics). If you only remember one thing from the rest of this guide, remember this: the KV cache is the single largest piece of state in LLM serving, and every serving optimization is ultimately about making it smaller, shareable, or pageable.

---

A short history of KV cache management

The KV cache as a concept is older than the modern LLM era. The KV cache as the dominant cost of inference is recent — it became the central concern only after long-context serving became mainstream. The optimizations that define 2026 production are mostly the work of the last three years. A timeline:

2017 — Vaswani et al., Attention Is All You Need. The original Transformer paper. Decoder attention reads K and V from previously-generated positions. The cache is implicit in the autoregressive formulation but isn't yet a discussed optimization concern — sequence lengths in the original paper are short (a few hundred tokens) and KV memory is trivial.

2018–2020 — GPT-1, GPT-2, GPT-3. The KV cache becomes operationally important as context windows grow to 1k, 2k, 4k. Implementations are still naive: contiguous per-sequence buffers, no sharing, no special memory management. KV memory is significant but still secondary to weight memory in most setups.

2019 — Shazeer, Fast Transformer Decoding (MQA, arXiv:1911.02150). First explicit acknowledgment that KV head count is a memory lever. Shazeer proposes Multi-Query Attention: one K and V head shared across all queries. The paper is mostly motivated by decode latency (less KV to read), not memory savings — but the memory benefit is the lasting impact.

2021 — Megatron-LM and tensor parallel KV. NVIDIA's Megatron-LM paper introduces the canonical pattern for splitting attention across GPUs along the head dimension. KV cache is implicitly sharded along the same axis. This is now the standard TP approach.

2022 — FlashAttention (Dao et al., arXiv:2205.14135). FlashAttention is primarily a compute optimization (kernel-level tiling, IO-awareness), not a KV management technique, but it has KV implications: by making attention dramatically faster, it shifts the bottleneck of long-context inference from compute to memory. The KV cache becomes the visible bottleneck.

*2023 (May) — Ainslie et al., GQA, arXiv:2305.13245.* The GQA paper proposes the middle ground between MHA and MQA: group multiple queries to share K and V. Llama-2 ships in July 2023 with GQA, kicking off the open-weight long-context era. The KV per token of a Llama-2 70B is 8× smaller than the (hypothetical) MHA equivalent. This single architectural decision changed the inference economics of every open-weight model that followed.

2023 (October) — Kwon et al., PagedAttention (vLLM, arXiv:2309.06180). The defining paper of modern KV management. Applies OS virtual-memory paging to KV. Eliminates internal and external fragmentation, lifts effective utilization from 30–50% to 90%+. Throughput on production-style workloads jumps 2–4× overnight. vLLM open-sources the implementation; within months, it's the most popular inference engine for open-weight models. See our LLM serving guide for how this changed the engine landscape.

2024 (early) — TGI, TRT-LLM, LMDeploy ship paged KV. The vLLM idea is general enough that other stacks adopt it within a few months. By mid-2024, paged KV is table stakes — any inference stack without it is positioned as legacy.

2024 (May) — DeepSeek-V2 with MLA (arXiv:2405.04434). DeepSeek introduces Multi-Head Latent Attention: K and V are projected into a low-rank latent before caching. Per-token KV drops to ~70 KB on a model the size of Llama-3 70B (vs 320 KB on standard GQA-8). The first architectural change since GQA that materially shifts KV economics. Adoption outside DeepSeek is slow because MLA requires custom kernels, but it sets a research direction. (Related: mixture-of-experts serving, where DeepSeek-V2's MoE structure intersects with its KV economics.)

2024 (June) — Zheng et al., SGLang (RadixAttention). SGLang generalizes block-level prefix sharing into a full radix tree keyed by token IDs. Cross-request, cross-session, cross-batch sharing through one mechanism. On chat-style traffic with shared system prompts, RadixAttention routinely doubles throughput vs vLLM-style block sharing.

2024 (June) — KIVI, KVQuant. Two papers (Liu et al., Hooper et al.) ship asymmetric per-channel-K, per-token-V calibration for INT4 KV quantization. Quality at 32k context becomes practical at INT4 for the first time. KVQuant's title — Towards 10 Million Context Length LLM Inference — captures the ambition.

2024 (October) — StreamingLLM with attention sinks (arXiv:2309.17453). Xiao et al. show that keeping the first 4 tokens (the "attention sinks") plus a sliding window of recent tokens preserves chat quality with bounded KV. Useful for long streaming dialogues; breaks retrieval at any context. A complementary eviction-by-attention-score line is H2O (Zhang et al., arXiv:2306.14048). See also long-context attention for how these techniques interact with retrieval workloads.

2025 (Q1) — FlashAttention-3 with paged KV native support. Previous FlashAttention versions had paged-attention support added on. FlashAttention-3 (Shah et al.) builds it in natively, with optimizations specific to Hopper and Blackwell. Paged attention performance closes the gap with contiguous attention.

2025 (mid) — EAGLE-2 standardizes speculative decoding. Li et al. publish EAGLE-2 with dynamic draft trees. The 2–3× speedup on agentic workloads is consistent enough that it ships as a default in vLLM, SGLang, and TRT-LLM. KV cache costs of speculative decoding (peak burden during verification) become a normal capacity-planning concern.

2025 (Q4) — Native Sparse Attention (NSA) from DeepSeek. Learned sparse attention competitive with dense at long contexts. Compute savings are large; KV memory is still mostly there but compressed. Pre-production as of early 2026 in some research stacks.

2026 (current) — Hybrid architectures in production. Jamba 52B (transformer/Mamba (arXiv:2312.00752) hybrid), some Gemma SWA hybrids, and a handful of pure-SSM variants are serving production traffic. KV cost categorically lower for these models. The cost ceiling on dense attention starts to show in product economics. See also disaggregated inference and the Mooncake KV-disaggregation report (arXiv:2407.00079) for how serving systems split KV-heavy and compute-heavy phases.

The lesson from this timeline: the KV cache problem isn't solved. Each year brings new techniques that change which costs dominate. The four-lever framework in this guide (architecture, quantization, paging, eviction) is the current synthesis, but it will look different in 2027 — likely with FP4, NSA, and architectural hybrids playing a much bigger role.

---

What is the KV cache?

To understand the KV cache, you need a sketch of attention.

In transformer attention, each layer takes the input hidden states for every token and projects them into three things: a query Q, a key K, and a value V. Attention then computes, for every token, a weighted sum of the values of all earlier (and same-position) tokens, where the weights are derived from Q · K.

In autoregressive decoding — generating one token at a time — you do this layer by layer for every new token. The token at position N attends to all tokens at positions 1 through N. If you naively recomputed K and V for all prior tokens at every step, you would do quadratic work per generated token. That is unacceptable past a few hundred tokens.

So we cache. After computing K and V for token N, we store them. When token N+1 arrives, we only compute its own K and V, append them to the cache, and reuse the previous N entries. Decode time per token becomes linear in context length, not quadratic. The cache that holds these K and V tensors across layers, heads, and tokens is what the field calls the KV cache.

The cache is per request. Two concurrent users have two completely separate KV caches; nothing mixes between them in the basic case. (Prefix caching, covered later, complicates this picture in a useful way.) The cache lives on the GPU because attention reads from it on every decode step — moving it off-GPU between steps would dwarf the actual compute.

The cache grows by one position per generated token. It does not shrink during a request. When a request finishes, its cache is freed. When the cache is full and a new request arrives, something has to give — that's the eviction problem covered later.

A historical note worth knowing: the KV cache concept predates the modern LLM era — it's standard in any autoregressive transformer (the original "Attention Is All You Need" decoder, GPT-1, GPT-2). What changed in 2023 onward is that the cache became the dominant memory cost of inference. Before billion-token contexts, weights dominated. After 32k+ context became routine, KV did. The mental model "weights are the cost" stopped being true around the time Llama-2 launched, and most engineers' intuition lagged the reality by about a year.

---

The math: deriving the per-token KV size

Let's build the formula component by component, because each piece teaches you something about the optimizations that follow.

Factor of 2. You cache both K and V. They have the same shape. Attention needs both: K to compute weights via Q · K, V to compute the weighted sum. There is no obvious way to derive one from the other without reintroducing quadratic compute, so both are stored. The factor of 2 is there in every variant.

num_layers. Each transformer layer has its own attention sub-block, with its own K and V. The cache is per-layer. A 32-layer model has 32× the cache of a 1-layer model with otherwise identical attention. This is why deep models (Llama-3 405B at 126 layers) have proportionally more KV than shallow ones.

num_kv_heads. The number of KV heads, not query heads. In Multi-Head Attention this equals the number of query heads. In Grouped-Query Attention it's smaller — typically 8 vs 64 query heads. In Multi-Query Attention it's exactly 1. This is the lever that GQA-style architectures pull, and it's a big lever — it's a multiplicative factor on the entire cache.

head_dim. The per-head dimension. Modern models almost universally use 128 (Llama family, Qwen, Mistral, DeepSeek). Some older or experimental models used 64 or 256. Don't assume — read the config.json. The hidden size is num_query_heads × head_dim, but the cache uses num_kv_heads × head_dim. They are not the same product when GQA is in play.

bytes_per_element. Set by the precision you store the cache in. BF16 = 2 bytes. FP16 = 2 bytes. FP8 = 1 byte. INT8 = 1 byte plus a small per-channel scale overhead. INT4 = 0.5 bytes plus larger overhead. FP4 (Blackwell only) = 0.5 bytes plus overhead. This is independent of the model's weight precision. You can have BF16 weights and FP8 KV, or FP8 weights and BF16 KV.

Putting it together for Llama-3 70B in BF16 with GQA-8:

2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KB per token

A 4096-token context costs 1.31 GB. A 32k-token context costs 10.5 GB. A 128k-token context costs 42 GB. A 1M-token context costs 328 GB — which is why nobody serves 1M-token contexts on Llama-3 70B without architectural tricks.

Now multiply by batch size. Eight concurrent Llama-3 70B requests at 4k context: 10.5 GB. Eight at 32k: 84 GB — already beyond an H100's 80 GB before you've put the model in. Eight at 128k: 336 GB.

This is why batch and context feel coupled. They are coupled. The KV cache is the product of both, and the product space gets unmanageable fast.

A note for MoE: only the active parameters are loaded for forward, but the KV cache is per layer regardless of expert routing. Mixtral 8×22B has 47B active parameters out of 141B total, but its KV cache cost is set by its 56 layers × 8 KV heads × 128 dim × 2 — same shape as if it were a dense 70B. MoE saves you on weight memory and FLOPs, not KV. This is one of the reasons MoE didn't immediately displace dense models for serving — the inference economics are different from the training economics.

---

Per-model worked examples

Concrete KV-per-token numbers for the major open and closed-weight models in 2026, all in BF16 unless noted:

Model	Layers	KV heads	head_dim	Bytes/elem	KV/token	32k context	128k context
Llama family
Llama-3 8B	32	8	128	2	128 KB	4.2 GB	16.8 GB
Llama-3 70B	80	8	128	2	320 KB	10.5 GB	42.0 GB
Llama-3 405B	126	8	128	2	504 KB	16.5 GB	66.1 GB
Llama-2 70B	80	8	128	2	320 KB	10.5 GB	42.0 GB
Llama-1 65B (MHA)	80	64	128	2	2.5 MB	84 GB	336 GB
Mistral / Mixtral
Mistral 7B	32	8	128	2	128 KB	4.2 GB	16.8 GB
Mixtral 8×7B	32	8	128	2	128 KB	4.2 GB	16.8 GB
Mixtral 8×22B	56	8	128	2	224 KB	7.3 GB	29.4 GB
Qwen
Qwen2.5 7B	28	4	128	2	56 KB	1.8 GB	7.3 GB
Qwen2.5 72B	80	8	128	2	320 KB	10.5 GB	42.0 GB
DeepSeek
DeepSeek-V2 (MLA)	60	n/a*	512 (latent)*	2	~70 KB	2.3 GB	9.2 GB
DeepSeek-V3 (MLA)	61	n/a*	512 (latent)*	2	~70 KB	2.3 GB	9.2 GB
Other
Falcon-180B (MHA)	80	64	128	2	2.5 MB	84 GB	336 GB
Phi-3 medium	40	10	128	2	200 KB	6.6 GB	26.2 GB
Gemma-2 27B	46	16	128	2	368 KB	12.0 GB	48.2 GB

* DeepSeek's MLA (Multi-Head Latent Attention) doesn't fit the standard formula because it caches a low-rank latent, not raw K and V. The effective per-token cost is roughly equivalent to the values shown.

A few patterns worth observing:

Llama-3 8B and Mistral 7B and Mixtral 8×7B all have identical KV per token. They share the same architectural shape (32 layers, 8 KV heads, 128 head_dim). For inference cost purposes, you can serve any of them with the same KV provisioning.

Qwen2.5 7B has a smaller KV than Llama-3 8B. Qwen uses 4 KV heads (GQA-7 effectively), giving it half the KV per token. This is a deliberate architectural choice that pays off in serving economics, especially at long context.

DeepSeek-V2/V3 with MLA is dramatically smaller per-token. ~70 KB vs 320 KB for a Llama-3 70B-class model. This is the headline benefit of MLA, and it's why DeepSeek can offer extremely cheap long-context API pricing — their KV economics are different from the rest of the field.

Llama-1 65B and Falcon-180B with MHA are categorically different. 2.5 MB per token. 32k context = 84 GB just for one request's KV. Long-context serving on these models was never economically viable; the GQA transition is what changed that.

For closed-weight models (GPT-4, Claude, Gemini), the architecture details are not public. From context-window pricing patterns and engineering interviews, it's reasonable to assume they all use GQA or something MLA-equivalent at the frontier. Closed-weight serving economics are fundamentally similar to open-weight; the math of the KV cache doesn't care about whether the weights leaked.

Concurrency math: how many simultaneous requests fit?

The capacity-planning question that matters: how many in-flight requests can one GPU hold? The formula:

max_concurrent = (HBM_bytes - model_weights - activation_workspace) / (avg_kv_per_request)

For Llama-3 70B FP8 on a single H100 SXM (80 GB):

• Model weights FP8: 70 GB
• Activation/workspace: ~2 GB
• Available for KV: ~8 GB
• At 8k context BF16 KV: 8 GB / 2.6 GB = ~3 concurrent requests.
• At 8k context FP8 KV: 8 GB / 1.3 GB = ~6 concurrent requests.
• At 32k context FP8 KV: 8 GB / 5.2 GB = ~1.5 concurrent.

On H200 (141 GB):

• Available for KV: ~69 GB.
• At 32k context FP8 KV: 69 GB / 5.2 GB = ~13 concurrent.
• At 128k context FP8 KV: 69 GB / 21 GB = ~3 concurrent.

On B200 (192 GB):

• Available for KV: ~120 GB after weights and workspace.
• At 128k context FP8 KV: ~5-6 concurrent.

The numbers above assume static allocation per request — actual paged-attention serving runs ~30-50% higher concurrency because most requests don't hit max context. The headline: single-H100 70B-class serving at long context is concurrency-starved. H200 doubles the practical concurrency; B200 doubles it again. The transition from H100 to H200 for long-context production serving was the dominant infra decision of 2024-2025.

Why DeepSeek's MLA changes the long-context economics

MLA caches a single low-rank latent vector per token instead of separate K and V vectors. For DeepSeek-V3 at 128k context, KV is ~9 GB per request — vs ~42 GB for a Llama-3 70B-class model. That single architectural choice means a single H100 can hold 7-8 concurrent DeepSeek-V3 requests at 128k context, where it could hold less than one Llama-3 70B request at the same context. This is also why DeepSeek's published API pricing at long context is 3-5x cheaper than equivalent-capability Llama serving — the underlying compute economics are categorically different. MLA's quality cost is mild (within 1-2 points on standard evals); the engineering cost is in training, not serving.

KV memory bandwidth, not just capacity

Per-step decode reads the entire KV cache for every layer. For a single request at 32k context on Llama-3 70B FP8 KV, that is 5.2 GB read per step. At 3 TB/s HBM bandwidth on H100, the KV read alone takes 5.2 / 3000 = 1.7 ms per step. At batch 32, it's 32 × 1.7 = 54 ms per step before any compute. This is why concurrency hits a ceiling well below memory-capacity-implied limits: at high batch, you become bandwidth-bound on KV reads. FP8 KV halves both capacity and bandwidth cost simultaneously, which is why it is essentially universal in production 2026.

---

Attention architecture: MHA, MQA, GQA, MLA

The KV head count is the lever, and the choice of attention variant is what determines it. Here's the full taxonomy as of 2026.

Multi-Head Attention (MHA)

The original. Each query head has its own dedicated K and V head. num_kv_heads = num_query_heads. Best quality, highest cost. Used in:

• Original Transformer (Vaswani et al., 2017)
• GPT-1, GPT-2, GPT-3
• Llama-1, Falcon-180B, Pythia, MPT
• Most pre-2023 models

Quality benchmark: the gold standard. Everything else is benchmarked against MHA.

Multi-Query Attention (MQA)

Shazeer's 2019 proposal: one K head and one V head shared across all query heads. num_kv_heads = 1. KV cache is num_query_heads× smaller than MHA, which is huge — 64× for a typical 64-query-head model. Decode latency drops because there's less K to read per attention computation.

Quality cost: visible. On standard benchmarks (MMLU, HellaSwag), the drop is ~1–3 percentage points vs MHA. On retrieval-style tasks (RULER, needle-in-haystack), the drop is larger — sometimes 5–10 percentage points at long context.

Used in:

• PaLM
• Falcon-7B, Falcon-40B
• Some research-scale models

MQA has largely been superseded by GQA in production. The quality cost was higher than the memory savings warranted for most use cases.

Grouped-Query Attention (GQA)

The compromise: multiple query heads share one K and V head, but in groups, not all together. Llama-3 70B has 64 query heads divided into 8 groups; each group of 8 query heads shares one K/V head. This is GQA-8.

The G in GQA-G is the number of KV heads, not the group size. So GQA-1 is MQA, GQA-N (where N = num_query_heads) is MHA, and GQA-8 is the standard middle ground.

Memory savings vs MHA: 8× for GQA-8. Quality cost: small. On Ainslie et al. (2023), GQA matches MHA within 0.5 percentage points on standard benchmarks. On retrieval, it's within 1–2 points — much better than MQA's 5–10.

Used in:

• Llama-2 (introduced GQA to open-weight)
• Llama-3, Llama-3.1, Llama-3.2, Llama-3.3
• Qwen2, Qwen2.5
• Mistral 7B, Mixtral
• Most 2024+ open-weight models

GQA-8 has become the de-facto standard. If you train a new transformer in 2026 without GQA, you should have a specific reason.

Multi-Head Latent Attention (MLA)

DeepSeek-V2's contribution (2024). Instead of caching K and V directly, MLA projects them into a low-rank latent of dimension d_latent, caches the latent, and reconstructs K and V on the fly during attention.

The cache becomes:

kv_bytes_per_token = num_layers × d_latent × bytes_per_element

Note: there's no factor of 2, no num_kv_heads, no head_dim. The latent is a single shared representation. For DeepSeek-V2 with d_latent=512 and 60 layers in BF16:

60 × 512 × 2 = 61,440 bytes = 60 KB per token

For comparison, GQA-8 on a similar-scale model would be ~240 KB per token. MLA is ~4× smaller again.

Quality: DeepSeek-V2 reports performance matching or exceeding their MHA baseline. The trick is that the rotary positional embedding interacts oddly with the latent projection, so MLA needs a special "decoupled rope" mechanism — the implementation is more complex than GQA.

MLA adoption outside DeepSeek has been slow. Reasons:

• Inference engines need MLA-specific kernels (mostly available in DeepSeek's own inference stack, partially in vLLM as of mid-2025).
• The architectural change is invasive — it's not a drop-in replacement for GQA.
• Training MLA from scratch requires its own tuning vs the well-trodden GQA path.

If MLA's quality and efficiency benefits hold across more architectures and the kernel support catches up, it's plausible MLA replaces GQA as the standard by 2027. But that's a forecast, not a fact.

Sliding-Window Attention (SWA)

Not exactly an attention head architecture but worth mentioning here: each token only attends to the previous W tokens. The KV cache caps at W tokens regardless of context. Used in Mistral-7B (W=4096), some Mistral variants, and as a layer pattern in hybrids.

Cost-wise: the cache is bounded. At 1M-token context, if W=4096, you only ever cache 4096 tokens. For applications that fit in a window (chat, code completion), this is enormous savings. For applications that need true long-range attention (long-document extraction, deep dependency tracking), SWA breaks because tokens past the window are not attended to.

Many production stacks now combine SWA layers with full-attention layers in a hybrid pattern (some Mistral variants, some Gemma variants), getting most of the efficiency with most of the long-range capability.

Sparse attention

Each token attends to a sparse subset rather than all prior tokens. The cache stays full but the effective attention compute drops.

Variants:

• Longformer's local + global attention (Beltagy et al., 2020): a sliding window plus a small set of "global" tokens.
• BigBird: window + random + global. Theoretical guarantees about expressivity.
• Native Sparse Attention (NSA) (DeepSeek, 2024–2025): learned sparsity. Each token learns which prior tokens to attend to.
• Reformer's LSH attention: hash-based sparsity. Underused in production but theoretically interesting.

Sparse attention reduces compute but not KV memory in most variants — you still cache all the K and V because you don't know in advance which will be sparsely attended to. The exception is some hierarchical sparse schemes that cache only at certain layers.

Sparse attention has not yet displaced dense attention in mainstream open-weight models. Whether NSA or its descendants change this is an open question.

Linear attention and SSMs

Architecturally distinct: Linear attention (Performer, Linear Transformer, RetNet, RWKV) and State Space Models (Mamba, Mamba-2) have no KV cache that grows with context. They have a fixed-size hidden state per sequence.

This is a categorically different cost structure. We'll cover this in Long-context architectures.

---

Quantizing the KV cache

The KV cache stores activations, not weights, so quantizing it is independent of quantizing the model. You can serve BF16 weights with FP8 KV, or FP8 weights with BF16 KV, or any combination.

The standard menu

Format	Bytes/elem	Quality cost vs BF16 (Llama-3 70B, MMLU/RULER 32k)	Stack support
BF16	2	baseline	every stack
FP16	2	~baseline (rounding diffs)	every stack
FP8 e4m3	1	~−0.1 / −0.5 pts	vLLM, SGLang, TRT-LLM, TGI
FP8 e5m2	1	~−0.2 / −0.7 pts	TRT-LLM, vLLM (legacy)
INT8 + scale	1	~−0.1 / −0.4 pts	TRT-LLM, vLLM (W8A16), llama.cpp
INT4 group	0.5 + overhead	~−0.3 / −2 to −5 pts on long-context retrieval	KIVI, KVQuant, llama.cpp Q4_K_M
FP4 (Blackwell)	0.5	~−0.5 / −1 to −3 pts (early data)	TRT-LLM (Blackwell only), early vLLM
Mixed (KIVI)	varies	best-in-class quality at INT2–INT4 average	KIVI implementation

The quality numbers above are typical from public KIVI / KVQuant evaluations and Meta's Llama-3 release notes. Your mileage varies with model and task.

FP8 e4m3 vs e5m2

Two FP8 formats exist on Hopper (H100/H200) and Blackwell (B200/GB200). They differ in how they trade exponent bits for mantissa bits:

• e4m3 (4 exponent, 3 mantissa, 1 sign): finer precision, narrower dynamic range. Good for activations and KV cache (typical activation values are in a moderate range).
• e5m2 (5 exponent, 2 mantissa, 1 sign): wider dynamic range, coarser precision. Good for gradients during training (gradients have wide dynamic range).

For KV cache, e4m3 wins. Use e4m3 unless you have a specific reason. Training pipelines may use e5m2 for activations during forward pass, but inference is e4m3.

Calibration: per-tensor, per-channel, per-token

Quantization needs scales — multipliers that map the float range to the integer range. The granularity of scales is a quality knob:

• Per-tensor: one scale for the entire cache. Fastest to apply, lowest memory overhead, lowest quality. The scale has to handle the most extreme value across the entire cache, so it's set conservatively and most values use only a fraction of the available range.
• Per-channel: one scale per head_dim slot. Each slot has its own typical magnitude, and per-channel scales let each be quantized to its own appropriate range. ~10% memory overhead for the scales themselves (one BF16 number per channel per layer per KV head). Modern default.
• Per-token: one scale per token position. Recovers more accuracy on outlier tokens. ~50% memory overhead vs per-channel. Used in some research setups, less common in production.

KIVI's contribution was finding the right asymmetry: per-channel for K, per-token for V. Empirically, K has outlier channels (some head_dim slots are systematically larger), while V has outlier tokens (some token positions have systematically larger values). Quantizing each in the granularity that matches its outliers preserves quality at lower bit widths than uniform schemes.

Practical implication: when you set --kv-cache-dtype fp8_e4m3 --calculate-kv-scales on vLLM, the calibration step computes per-channel scales using a calibration set. If you skip calibration, vLLM falls back to per-tensor scales and you'll see 1–2 extra points of quality loss on RULER. Don't skip calibration.

When INT4 KV breaks

INT4 KV (KIVI, KVQuant, llama.cpp Q4_K) breaks long-context retrieval before it breaks short-context generation.

Why: attention over many positions accumulates many small dot-products. Each dot-product has small numerical error from INT4 quantization. The errors compound. At 4k context, the accumulated error is small. At 128k context, the accumulated error can shift attention weights enough that the model attends to the wrong positions.

The empirical pattern, from KIVI / KVQuant evaluations:

• 4k context, MMLU: INT4 KV loses 0.3 points vs BF16. Negligible.
• 32k context, RULER: INT4 KV loses 2 points vs BF16. Noticeable but acceptable for many tasks.
• 64k context, RULER: INT4 KV loses 5 points. Visible degradation.
• 128k context, RULER (needle-in-haystack): INT4 KV loses 10+ points. Often unusable for retrieval.

If your workload is summarization, chat, or short-form generation, INT4 KV is fine. If it is long-document extraction, RAG with deep retrieval, or code with cross-file dependency tracking, stick with FP8 KV (or BF16 if you have the headroom).

FP4 on Blackwell

Blackwell (B200, GB200) introduces native FP4 support. KV cache in FP4 is half the size of FP8 again — quartile of BF16 — and Blackwell's tensor cores compute FP4 GEMMs at very high throughput.

Quality data is preliminary. Early TRT-LLM benchmarks (Q4 2025) show FP4 KV losing 1–3 points on RULER 64k vs FP8, depending on model. Expect this to improve with better calibration techniques over 2026.

For now: FP4 KV is interesting for B200-only deployments where you need to fit very long contexts. It's not yet the default. Watch for KIVI-style asymmetric calibration techniques to land for FP4 in 2026 — that's likely to make FP4 production-viable.

Enabling KV quantization in practice

vLLM (most common):

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --kv-cache-dtype fp8_e4m3 \
  --calculate-kv-scales \
  --max-model-len 131072

SGLang:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.3-70B-Instruct \
  --kv-cache-dtype fp8_e4m3

TensorRT-LLM (Hopper FP8):

trtllm-build --checkpoint_dir ./ckpt \
  --use_fp8_context_fmha enable \
  --paged_kv_cache enable \
  --max_input_len 131072

LMDeploy:

lmdeploy serve api_server meta-llama/Llama-3.3-70B-Instruct \
  --quant-policy 8 \
  --session-len 131072

llama.cpp (community-maintained, mostly INT4):

./llama-server \
  -m models/llama-3-70b.q4_k_m.gguf \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  -c 32768

In every stack, the equivalent flag exists. The exact name and format vary. Read the docs for your version.

---

PagedAttention: the OS-style memory manager

PagedAttention (Kwon et al., 2023) was the most impactful inference-systems paper of the past five years. It applied the operating-systems idea of virtual memory paging to KV cache management.

The problem before paging

Naive KV management allocates one contiguous buffer per sequence, sized to max_context_length. This produces two kinds of waste:

Internal fragmentation. You allocated 32k tokens of buffer; the request only generates 2k tokens. The other 30k tokens of buffer is wasted for the duration of the request. On a workload where most requests are short, internal fragmentation is the dominant waste.

External fragmentation. Sequence A finishes after 20k tokens, freeing 32k. Sequence B finishes after 5k tokens, freeing 32k somewhere else. Now you want to allocate a 40k buffer for sequence C — but neither freed region is large enough, even though the total free memory exceeds 40k. External fragmentation is the dominant waste on long-running, mixed-length workloads.

Combined, naive allocation typically uses 30–50% of the KV memory you've nominally allocated. The other 50–70% is wasted to fragmentation.

The paging insight

Kwon et al. observed that this is exactly the problem operating systems solved decades ago with virtual memory: divide memory into fixed-size blocks (pages), maintain a per-process page table, and let pages be allocated wherever there's free space. The OS doesn't require the physical pages backing a process's address space to be contiguous; it just maintains the mapping.

PagedAttention applies the same trick to KV. The KV cache is divided into fixed-size blocks (default 16 tokens). Each sequence has a block table mapping its logical positions to physical block IDs. When a sequence needs more KV space, the system allocates an arbitrary free block — no contiguity requirement. When a sequence finishes, its blocks are freed and immediately reusable.

Concrete utilization numbers

From the vLLM paper and independent reproductions on production traces:

• Naive contiguous allocation: 30–50% effective KV utilization.
• Paged with block_size=16: 90–96% utilization. Internal fragmentation is bounded by the block size (~16 tokens of waste at the tail of each sequence). External fragmentation goes to zero.

That delta is 2–3× the in-flight requests at the same memory budget. Paging is now the default in vLLM, SGLang, TRT-LLM, TGI, LMDeploy, and llama.cpp's server.

If you are evaluating a serving stack in 2026 and it doesn't page the KV cache, that is a hard pass.

Block size: the one knob worth tuning

vLLM (and most stacks) accepts block_size ∈ {8, 16, 32, 64, 128}. Tradeoffs:

• block_size=8: lowest tail waste (~8 tokens/sequence on average), more block-table memory, higher attention-kernel overhead per block lookup.
• block_size=16: the default, the sweet spot for most workloads.
• block_size=32 or 64: better attention kernel locality (fewer block-table indirections per attention computation), more tail waste, helps when most sequences are >2k tokens (long-context-heavy serving).
• block_size=128: experimental; only used in some research configs. Tail waste is real (up to 128 tokens) but kernels can be very efficient.

Most teams should leave it at 16. If you serve mostly long requests (RAG, document analysis, code), test 32 — it sometimes wins on throughput by ~5%. If you serve mostly short requests (chatbots, classification), 8 might marginally help, but probably not enough to matter.

Implementation cost: attention kernels need to be paged-aware

A subtlety: when KV is paged, attention kernels can't just read a contiguous K and V buffer — they have to follow the block table. This changes the kernel implementation.

vLLM's original paper introduced custom CUDA kernels for paged attention. Later, FlashAttention-2 and FlashAttention-3 added paged-attention support, and these are the default in most stacks now.

The performance cost of paging vs contiguous: ~5–10% on attention compute, depending on block size and sequence length. The memory savings (2–3× more in-flight requests) trivially dominate this overhead.

Paged KV in serving stack timelines

• vLLM v0.1 (2023): introduced PagedAttention. KV memory utilization went from ~40% to ~94% overnight. Throughput on production-style workloads jumped 2–4×.
• TRT-LLM (2023): shipped paged KV with NVIDIA-internal kernels.
• SGLang (2024): built on paged KV, added RadixAttention prefix sharing on top.
• TGI (Hugging Face, 2024): added paged attention.
• LMDeploy (2024): paged from inception, focused on Chinese open-weight models.
• llama.cpp server (2024): added KV cache management (less aggressive paging, but conceptually similar).

By 2026, paged KV is table stakes. If a stack you're considering doesn't page, ask why.

How paged attention kernels actually work

A practical aside on what's happening at the kernel level when you read "paged attention." The mental model is straightforward but the implementation has subtleties worth knowing if you ever profile or debug attention.

In contiguous attention, computing attention for a single query against N cached positions looks roughly like:

1. Load Q (one query vector) into shared memory.
2. Loop over N positions in the K cache:
   a. Load K[i] into shared memory.
   b. Compute score[i] = Q · K[i].
3. Compute softmax(scores).
4. Loop again:
   a. Load V[i] into shared memory.
   b. Accumulate weighted V[i].
5. Write out attention output.

K and V are contiguous in memory, so loads are coalesced (consecutive threads read consecutive addresses). Hardware prefetching works well. Throughput is bounded by HBM bandwidth — typically 2–3 TB/s on H100, ~5 TB/s on H200, ~8 TB/s on B200.

In paged attention, K and V for a single sequence are scattered across multiple physical blocks. The same loop becomes:

1. Load Q into shared memory.
2. Look up the sequence's block table (a small array on GPU).
3. For each logical block in the sequence:
   a. Use block_table[i] to get the physical block address.
   b. Load that physical block's K into shared memory.
   c. Compute scores for the tokens in that block.
4. Softmax across all blocks.
5. Repeat for V.
6. Write output.

The new ingredient is the block-table indirection. Each load now requires two memory accesses: one for the block table (small, often cached) and one for the actual K or V data.

The cost of paging at the kernel level.

Naively, this should be slow — every load chases a pointer. In practice, the cost is small (~5–10% on attention compute) for several reasons:

1. Block tables are tiny and stay in L2/L1 cache. A sequence of N tokens with block_size=16 has N/16 entries in its block table — for a 32k sequence, that's 2000 entries × 4 bytes = 8 KB. It fits comfortably in cache.

2. Within a block, reads are still contiguous. All 16 tokens of a block are stored sequentially in physical memory. The expensive part — actually reading K and V — is not random; only the block-to-block transitions involve indirection.

3. The block table can be loaded once per attention computation, not once per token. A clever kernel reads the entire block table into shared memory at the start, then proceeds without further indirection.

4. Modern attention kernels (FlashAttention-3) bake paged-aware logic into the inner loop, fusing block-table lookups with the existing tile loops. The performance overhead of paged vs contiguous becomes barely measurable.

FlashAttention-3 specifics on Blackwell.

FlashAttention-3 (Shah et al., 2024) adds two things relevant to KV cache:

• Async memory loads. Hopper and Blackwell support TMA (Tensor Memory Accelerator) for asynchronous loads. While one block of K is being computed against, the next block is being loaded. The compute and memory operations overlap, hiding the latency of even the indirect loads.

• Native FP8/FP4 attention. Computing the QK matrix product in FP8 directly (instead of dequantizing to BF16) doubles throughput on Hopper, quadruples it on Blackwell with FP4. Combined with paging, the kernel handles per-block scales automatically for paged-and-quantized KV caches.

The practical implication: on a current Hopper or Blackwell GPU with FlashAttention-3 and paged KV, attention is ~no slower than it would be with contiguous KV. The "paged is slower" intuition from 2023 is outdated.

Where paged kernels still trail contiguous.

Two cases:

• Very short sequences (≤256 tokens) where the block-table overhead is a meaningful fraction of total work. For these, kernels often fall back to a contiguous path.
• Variable-length batches where some sequences are 4k and others are 32k. The kernel has to handle many different block-table sizes per warp. FlashAttention-3 handles this via "split-K" dispatch but there's still a small overhead vs uniform-length batches.

For the dominant production case (medium-to-long contexts, mixed batch sizes), paging is essentially free at the kernel level.

---

Prefix caching and RadixAttention

Paging unlocks the next-level optimization: block-level prefix sharing. When two requests have the same prefix tokens, they can share the same KV blocks until they diverge.

The setup

In production LLM serving, requests often share substantial prefixes:

• A multi-tenant chat product has a fixed system prompt prepended to every user message.
• A RAG application retrieves passages and prepends them to the query.
• A code completion tool sends the editor state plus a stable prompt.
• Multi-turn chat: every turn shares all prior turns of the conversation history.

If each request computes KV from scratch for the shared prefix, you are wasting compute and memory on a deterministic input. Prefix caching dedups this.

Block-level prefix sharing in vLLM

vLLM's --enable-prefix-caching (default in v0.6.0+) implements block-level dedup: when a new request arrives, vLLM checks if the prefix's blocks are already in the cache (hashed by token IDs). If so, the request reuses those blocks without recomputing.

The hash is per block. Two requests with prefix tokens [1, 2, 3, ..., 32] (two 16-token blocks) and a third with prefix [1, 2, 3, ..., 16, 99, ...] will share the first block but not the second. Sharing is position-based and token-based: the first 16 tokens have to be byte-identical for the first block to be reused.

RadixAttention: SGLang's more aggressive sharing

RadixAttention (Zheng et al., 2024) is SGLang's contribution. It generalizes block-level prefix sharing to a full radix tree.

The mechanism:

• Every distinct prefix in the cache is represented as a node in a radix tree, keyed by token IDs.
• When a new request arrives, the tree is traversed to find the longest matching prefix.
• The request mounts at the matched node and only computes KV for the remaining suffix.
• When a request finishes, its leaf node may be evicted (LRU), but shared internal nodes are protected as long as any descendant is alive.

Effects vs vLLM-style block-level caching:

• Cross-session sharing: if the same prefix appears in requests across different sessions, RadixAttention shares them. Block-level caching does too if the cache is global, which is the usual configuration.
• Cross-batch sharing: even within one batch, two requests with the same prefix share blocks.
• Eviction is smarter: shared internal nodes are protected from eviction even when their original requesting sequences are long gone, because new sequences mounted on them.

In practice, both vLLM's block-level and SGLang's RadixAttention deliver 2–10× throughput improvements on workloads with significant prefix sharing. RadixAttention has a slight edge on highly varied workloads with deep prefix trees.

Real workload numbers

From public LMSys / vLLM measurements:

• Chat with 1k-token system prompt, 100 concurrent users: ~95% prefix hit rate. Effective KV usage drops 4–6× vs no sharing.
• RAG with 8k retrieved-context per request, low overlap between queries: prefix hit rate drops to 10–30%, depending on retrieval determinism.
• Code completion with editor-state prefix: 80–90% hit rate within a session.
• Multi-turn chat with conversation history growing across turns: essentially 100% — every turn shares the prior turn's KV.
• Few-shot prompting (consistent k-shot examples): 85–95% hit rate if examples are stable.

If your workload has a long fixed system prompt, RadixAttention-style sharing routinely beats every other optimization on this list. Run the measurement first.

What breaks prefix caching

Prefix caching depends on stable prefix tokens. The things that break it, often invisibly:

• Adding timestamps to system prompts ("It is currently May 7, 2026..."). Every request has a different prefix.
• Embedding session IDs in prompts. Same problem.
• Random sampling temperature shifts. Doesn't break the cache itself, but if you re-prompt with a slightly different system prompt for different temperatures, you lose the share.
• Tokenizer changes mid-deployment. Cached blocks reference token IDs from the old tokenizer.

If your prefix cache hit rate is mysteriously low on a workload that should share, look for these patterns first.

Eviction strategies for the prefix cache

When the cache is full and a new prefix needs to land, something has to go. Common strategies:

• LRU (default): evict the least recently used leaf. Works well for chat-like access patterns.
• LFU: evict the least frequently used. Better for workloads with stable hot prefixes.
• TTL: evict after a fixed time. Simple, predictable, sometimes wasteful.

vLLM uses LRU. SGLang uses LRU at the leaf level with internal-node protection. Most users don't need to tune this; the defaults are reasonable.

---

Multi-GPU: tensor parallelism and KV sharding

When the model doesn't fit on one GPU, you split it across many. The KV cache splits too, but in a specific way.

Tensor parallelism (TP) and KV head sharding

Tensor parallelism splits each weight matrix (specifically the projection matrices in attention and MLP) across N GPUs. For attention specifically, the K and V projections are split along the head dimension: with TP=N, each GPU holds 1/N of the KV heads.

For Llama-3 70B (8 KV heads) at TP=2:

kv_per_gpu_per_token = 2 × 80 × 4 × 128 × 2 = 160 KB

Each GPU holds half the KV cache. Total system KV is unchanged; per-GPU KV is halved. This is one of the major reasons people run TP=4 or TP=8 on H100s for large models — it linearly drops the per-GPU KV burden.

A subtlety: TP only divides the KV cache cleanly when num_kv_heads is divisible by TP degree. Llama-3's 8 KV heads support TP=1, 2, 4, or 8 cleanly. Models with num_kv_heads=2 (some MQA-adjacent configs) cap at TP=2. Models with num_kv_heads=1 (pure MQA) can't TP the attention at all — though some implementations replicate the single KV head across TP ranks, paying memory to gain compute parallelism.

Pipeline parallelism (PP) and KV by layer

Pipeline parallelism puts different layers on different GPUs. KV cost per GPU is just (layers_on_this_gpu / total_layers) × full_cache.

For Llama-3 70B (80 layers) at PP=4:

kv_per_gpu_per_token = 2 × 20 × 8 × 128 × 2 = 80 KB

A 32k context request spreads across 4 GPUs at 80 KB × 32k = 2.6 GB per GPU.

PP is rarely used alone for inference because it serializes — a token has to flow through GPU 0, then 1, then 2, then 3 to be generated, and idle bubbles are hard to fill. Most production stacks combine PP with TP (e.g. TP=2 × PP=4 for big models on 8 GPUs).

Expert parallelism (EP) for MoE

For MoE models like Mixtral, expert parallelism (EP) puts different experts on different GPUs. KV cache is unaffected by EP — KV is per layer, not per expert. EP sharding shows up in MLP routing, not in attention.

Combined parallelism

Modern serving stacks let you mix TP, PP, and (for MoE) EP. The thing to verify when planning:

total_kv = (kv_heads / tp) × (layers / pp) × head_dim × 2 × bytes × ctx × batch

needs to fit on each GPU. Make sure this fits per GPU under your chosen TP/PP, not just in aggregate.

vLLM, SGLang, TRT-LLM, and LMDeploy all handle TP+PP automatically. You set:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2

and they split. The thing that breaks: subtle interactions between paged KV and PP boundaries, especially on the first or last layer of a pipeline stage. Most stack versions handle this; verify with your specific version.

NCCL communication and the cost of TP

Tensor parallelism isn't free. Every TP-parallel layer requires an all-reduce (or all-gather + reduce-scatter) to synchronize activations across GPUs. The cost depends on activation size and inter-GPU bandwidth.

For a typical LLM forward pass at TP=N:

• 2 all-reduces per transformer layer (one after attention, one after MLP).
• Each all-reduce moves batch × seq_len × hidden_size × bytes_per_element bytes per direction.

For Llama-3 70B at TP=4 (hidden_size=8192) with batch=8 at 2k context, BF16:

• One all-reduce moves 8 × 2048 × 8192 × 2 = 256 MB.
• 80 layers × 2 all-reduces = 160 all-reduces per forward pass.
• Total intra-step communication: ~40 GB.

NVLink bandwidth on H100 is 900 GB/s/GPU (NVLink 4). All-reduce overhead at TP=4 is roughly 40 GB / (900 GB/s × 2) ≈ 22 ms — a meaningful fraction of decode latency.

The reason TP=4 still wins despite this overhead: the per-GPU compute drops 4×, the per-GPU KV drops 4×, and good schedulers overlap the communication with the next layer's compute. The net is positive on H100 with NVLink. Without NVLink (PCIe-only multi-GPU), the picture changes — TP becomes much more expensive.

The practical limit: TP=8 within a node is fine on NVLink-connected GPUs (DGX H100, B200 NVL72). TP=16 across nodes requires InfiniBand or RoCE and the all-reduce starts to dominate. Most production deployments cap TP at 8.

Async compute/communication overlap

Modern serving stacks overlap the all-reduce of layer N with the compute of layer N+1. The pattern:

1. Compute layer N's attention output.
2. Kick off the all-reduce. (Async, non-blocking.)
3. Start computing layer N's MLP on the partial output.
4. When the all-reduce finishes, finalize the MLP and move to layer N+1.

In practice, this hides 40–70% of communication latency. The visible cost of TP shrinks accordingly.

This is why TP=4 isn't 4× slower than TP=1 from a wall-clock perspective — typically it's 1.5–2.5× faster per token (4× compute + KV split, minus 1.5× communication overhead).

Expert parallelism (EP) for MoE models

Mixture-of-Experts models like Mixtral 8×22B, DeepSeek-V2, and others have distinct expert MLP blocks per layer. Expert parallelism distributes experts across GPUs.

KV cache is not affected by EP — KV is per layer per token, not per expert. EP changes the MLP routing but doesn't touch attention.

What EP does affect: communication. Every token needs to be routed to its assigned experts and back. This is an all-to-all communication, not an all-reduce. All-to-all on NVLink is faster than across nodes; for production EP, all experts on one node is the norm.

A specific number: DeepSeek-V2 with EP=8 (one expert per GPU on a DGX node) sees ~30% extra latency vs single-expert-per-GPU baselines, dominated by the all-to-all. The compute savings (only 21B active parameters per token) more than compensate.

When you serve MoE models, you typically combine TP+EP. Llama-4 Maverick (hypothetical 2026 architecture) at TP=2 × EP=4 on 8 GPUs: each GPU holds 1/2 of attention and 1/4 of experts. KV is split via TP only.

Sequence parallelism (SP) and context parallelism (CP)

For very-long-context inference, even TP+PP may not be enough. Two newer sharding strategies:

Sequence parallelism: split the sequence dimension across GPUs in addition to the head/layer dimensions. Each GPU computes attention for a contiguous chunk of tokens. Used in Megatron-LM and increasingly in inference for >256k contexts.

Context parallelism / Ring attention: split the sequence such that each GPU holds part of the K and V, and a ring-style communication passes K/V chunks around. Used in Gemini's purported 1M+ context inference and some research-grade serving setups.

KV implications: with SP/CP, the per-GPU KV is total_kv / N for sequence dimension N. This unlocks contexts that wouldn't fit on any single GPU's HBM, even with TP.

Neither SP nor CP is widely deployed in 2026 open-source serving stacks for inference (training stacks like Megatron-LM use them extensively). Watch for vLLM and SGLang to add CP support over 2026 if 1M+ context becomes a serious production target.

When to add a GPU vs add a replica

A common decision: you're at capacity. Should you scale up (more GPUs per replica, larger TP) or scale out (more replicas, same TP)?

Scale up (add a GPU to TP) when:

• You're memory-bound and a single request doesn't fit at current TP.
• You're serving very-long-context with low concurrency (1 request at 256k context).
• You have NVLink-connected hardware that makes higher TP cheap.

Scale out (add a replica) when:

• You're throughput-bound and individual requests fit fine.
• Your workload has high concurrency, modest context.
• You want failure isolation (one replica down ≠ service down).

The break-even rule of thumb: if TP=N supports concurrency C, scaling out gives you C concurrency per replica with linear scale. Scaling up gives you >C concurrency on one replica but with diminishing returns on the TP overhead. For most chat/agent workloads at 32k context, scale out wins beyond TP=2 or TP=4.

For long-context-heavy workloads (RAG with 64k+ inputs, document analysis), scaling up keeps winning longer because each request needs significant memory.

NUMA and PCIe topology gotchas

A subtle production issue: on a multi-socket server, GPUs may be attached to different CPU sockets via different PCIe root complexes. CPU-to-GPU bandwidth depends on which socket the inference process is pinned to.

Symptom: random 1.5–2× variance in inference latency that doesn't correlate with anything obvious in the workload.

Fix: pin processes to specific NUMA nodes (numactl --cpunodebind=0 --membind=0). Use nvidia-smi topo -m to see GPU-to-CPU topology. For multi-replica servers, ensure replicas don't compete for the same NUMA node.

This is a 2010s-era concern that resurfaced because LLM inference makes heavy use of pinned host memory (CPU offload, prefix-cache lookups). On a clean NVLink-only setup, NUMA matters less. On servers where some GPUs sit behind a CPU PCIe controller, it matters a lot.

---

Offloading: CPU, NVMe, hierarchical KV

When even paged FP8 KV doesn't fit, the next move is to push some of the cache off the GPU.

CPU offload

Inactive sequences (waiting in queue, paused, or low-priority) live in pinned host memory; the active set lives in HBM. When a paused sequence needs to resume, its KV is swapped back to HBM.

Bandwidth math:

• PCIe Gen5 x16: ~64 GB/s/direction. Each direction is independent.
• PCIe Gen4 x16: ~32 GB/s/direction.
• NVLink (within a node): 600+ GB/s on H100 and successors. Used for GPU-to-GPU, not GPU-to-host.

For a 1 GB sequence's KV cache:

• PCIe Gen5: 1 GB / 64 GB/s = 16 ms swap-in.
• PCIe Gen4: 1 GB / 32 GB/s = 32 ms swap-in.

Cheap if swaps are rare. Ruinous if every request causes one.

NVIDIA's TensorRT-LLM v0.13+ exposes --enable-cpu-offload. vLLM has experimental CPU offload via --cpu-offload-gb. SGLang doesn't currently support CPU offload as of mid-2025 (verify in current docs).

NVMe / hierarchical KV

Cold prefix blocks spill to local NVMe SSDs. Bandwidth math:

• PCIe Gen5 NVMe: ~7 GB/s read on consumer SSDs, up to 14 GB/s on enterprise.
• PCIe Gen4 NVMe: ~3.5 GB/s on consumer, 7 GB/s on enterprise.

For a 10 GB hierarchical KV reload:

• Gen5 enterprise NVMe: 10/14 = 0.7 s.
• Gen5 consumer: 1.4 s.

This is far too slow for interactive serving. For batch retrieval and analytical workloads where the cache is mostly cold, it's tolerable.

The 1M-token context demos you've seen (Gemini 1.5 Pro, some Claude variants, GPT-4 Turbo at 128k+) almost certainly involve hierarchical storage or attention sparsity behind the scenes. Pure dense attention with full KV at 1M context on commodity GPUs is not free.

When to offload, when not to

Offload when:

• You have abundant host memory (CPU offload) or NVMe (hierarchical) and can tolerate occasional latency spikes.
• Your workload has very long contexts but cold cache patterns (RAG with diverse queries, batch document processing).
• The throughput cost of not offloading (smaller batches, more eviction) exceeds the latency cost of swaps.

Don't offload when:

• You're serving interactive chat with strict P95 latency SLAs.
• Your cache patterns are hot (every request touches every block).
• You can afford more GPUs to stay GPU-resident.

For interactive serving, GPU-resident is still the default. Offload is a niche optimization for specific access patterns, not a free win.

---

Eviction strategies when the cache fills

Every serving stack hits this eventually. New request arrives, no free blocks. What now?

The options, ranked by what production stacks actually use

1. Reject until capacity frees. Used as a last resort or behind an HTTP 429. Punishes throughput but is sometimes the correct choice if downstream guarantees matter.

2. Preempt + recompute. Evict the KV of an in-progress sequence, restart its prefill when it gets rescheduled. vLLM's default. Cost: 1× extra prefill per evicted sequence. The recompute cost is bounded — you only pay for what's already been generated, which is shorter than the full eventual sequence.

3. Preempt + swap. Move the evicted sequence's KV to host memory (CPU offload pattern), bring it back when rescheduled. Faster on resume than recompute (no extra prefill), more complex implementation, depends on PCIe bandwidth.

4. Compress further. Switch a sequence to INT4 KV mid-stream to free space. Experimental — KIVI's online quantization, MiKV. Quality drops mid-stream are visible; rarely shipped in production.

5. Drop tokens (StreamingLLM / attention sinks). Keep the first 4 tokens (the "attention sinks" that softmax disproportionately puts weight on) plus the last N tokens. Discard everything in between. Works surprisingly well for chat (Xiao et al., 2024). Breaks retrieval at any context length — you can't attend to a passage that's been dropped.

vLLM's default: recomputation-based preemption

vLLM uses option 2. When the cache fills:

1. Pick the lowest-priority sequence (longest-running, lowest probability of timely completion).
2. Evict its KV blocks.
3. Add it to the recompute queue.
4. When capacity frees, reschedule it: prefill its current full context from scratch.

Recompute cost is real — a sequence that's already generated 5k tokens has to redo all 5k tokens of prefill on reschedule. But it's a bounded cost, and it's strictly better than refusing the request entirely.

SGLang's approach: aggressive sharing avoids the problem

SGLang's RadixAttention often dodges the eviction question entirely on chat-style traffic. Because so many requests share prefixes, the effective KV demand is much lower than the naive estimate would suggest. By the time you'd hit eviction in a non-RadixAttention stack, RadixAttention is still at 60% utilization.

The corollary: if your workload doesn't have prefix sharing (highly varied prompts), RadixAttention's advantage shrinks. It's a workload-specific optimization.

When you have to tune eviction

Most teams should never tune eviction policy. The defaults are right. You should think about it when:

• You see eviction_rate > 5/sec in your metrics for sustained periods.
• Your P95 latency has fat tails caused by recompute on preempted sequences.
• You're operating at >90% sustained KV utilization and adding more replicas isn't the answer.

In those cases, switch to swap-based preemption (if your stack supports it and you have PCIe headroom) or reduce max_num_seqs (admit fewer concurrent requests, queue more aggressively at the API layer).

---

KV cache and speculative decoding

Speculative decoding generates K candidate tokens with a small "draft" model, then verifies them in one pass through the large "target" model. If the candidates are accepted, you got K tokens for the cost of about 1 target-model forward pass plus K cheap draft-model passes. If they're rejected, you fall back to the standard one-token-per-pass.

The KV cache implications are non-obvious.

Two separate caches

The draft model has its own KV cache; the target model has its own. They are not shared — they're different models with different architectures.

Draft model is typically 7B or smaller; target is 70B+. Even though the draft is much smaller, its KV cache adds memory.

For Llama-3 70B target + Llama-3 8B draft at 32k context:

• Target KV: 10.5 GB per request.
• Draft KV: 4.2 GB per request.
• Combined: 14.7 GB per request, vs 10.5 GB for non-speculative.

That's 40% more KV memory per concurrent request. You have to size for it.

Per-step memory peaks

Spec-decode generates K candidates per step. The target model processes K positions in one forward pass. So the target's KV grows by K per step, not 1.

If K=4 (typical EAGLE-2 setup), each verification step writes 4 new KV positions to the target cache. The cache grows 4× faster per wall-clock unit than non-speculative decode would.

Why this matters: peak KV usage during a 32k-context sequence isn't just 32k × kv_per_token; it's 32k × kv_per_token plus the draft's contribution. Capacity planning has to account for the peak, not the average.

EAGLE, MEDUSA, and the 2026 standard

By 2026, three speculative decoding approaches dominate:

• EAGLE-2 (Li et al., 2024): uses a small "head" model that shares the target's hidden states. Draft model is essentially free in compute but has its own KV-like state.
• MEDUSA (Cai et al., 2024): adds multiple decoding heads to the target model itself, predicting K tokens in parallel. No separate draft model. KV is the target's only.
• Lookahead decoding (Fu et al., 2024): the target model itself drafts and verifies via lookahead. No draft. Less aggressive speedup but no extra KV.

Throughput wins (typical, on Llama-3 70B):

• EAGLE-2: 2–3× on agentic workloads, 1.3–1.8× on chat.
• MEDUSA: 1.5–2× on most workloads.
• Lookahead: 1.2–1.5×.

EAGLE-2 has the highest peak speedup but the highest KV cost. MEDUSA has lower peak but no extra KV. Choose based on whether your bottleneck is compute or memory.

Tuning draft length K

K too short (e.g., K=2): verification doesn't amortize. The savings from accepting K tokens at once are too small.

K too long (e.g., K=8+): draft accuracy drops. You reject most candidates and pay for the draft compute and KV without benefit.

Sweet spot: K=4 for most workloads. K=6–7 for highly predictable workloads (code completion, structured output). K=2–3 for highly variable workloads (creative writing, reasoning).

Some stacks (vLLM, TRT-LLM) auto-tune K based on observed acceptance rates. This is generally better than hand-tuning.

---

Long-context attention: SWA, sparse, linear, SSM, hybrid

For million-token contexts, dense attention with a full KV cache is physically infeasible on commodity hardware. Several architectural approaches change the equation rather than optimize the storage.

Sliding-window attention (SWA)

Each token attends to the previous W tokens. KV cache caps at W tokens regardless of context. Used in Mistral-7B (W=4096), GPT-OSS-120B variants, some Gemma variants.

Pros: cache is bounded. 1M-token context with W=4096 only ever caches 4096 tokens. 245× less memory than dense at the same context.

Cons: tokens past the window are unrecoverable. No long-range attention. SWA is wrong for any task that requires referencing positions outside the window.

Sweet spot: chat, code completion, anywhere local context dominates. Many production deployments combine SWA layers with full-attention layers in a hybrid pattern, getting most of the efficiency with most of the long-range capability.

Sparse attention

Each token attends to a sparse subset rather than all prior tokens.

Variants:

• Longformer (Beltagy et al., 2020): local sliding window plus a small set of "global" tokens.
• BigBird (Zaheer et al., 2020): local window + random + global. Theoretical guarantees about expressivity.
• Native Sparse Attention (NSA) (DeepSeek, 2024–2025): learned sparsity. Each token learns which prior tokens to attend to.
• Reformer's LSH attention (Kitaev et al., 2020): hash-based sparsity.

In most variants, sparse attention reduces compute but not KV memory — you still cache all K and V because you don't know in advance which will be sparsely attended to. Some hierarchical sparse schemes cache only at certain layers, partially reducing memory.

NSA is the most production-relevant 2026 variant. DeepSeek reports that NSA on long-context benchmarks matches or exceeds dense attention while using ~1/4 the compute. KV memory is unchanged.

Linear attention and SSMs

These architectures replace the QK-softmax attention with mechanisms that have no KV cache that grows with context. Instead, they have a fixed-size hidden state per sequence.

Linear attention (Performer, Linear Transformer, RetNet): rewrites attention as a kernel approximation that allows computation in O(N) instead of O(N²), and admits a recurrent formulation with constant per-step memory.

State Space Models (SSMs): Mamba, Mamba-2 (Dao & Gu, 2024). A fundamentally different architecture using selective state-space recurrences. Train in parallel like a transformer, run with fixed-state recurrence at inference.

RWKV (Peng et al., 2023): a custom RNN architecture trained transformer-style. Constant memory at inference.

The cost difference is categorical, not incremental. A 7B parameter SSM uses the same per-step memory at 1M-token context as at 1k-token context. A transformer uses 1000× more.

The catch: pure SSMs have weaker in-context learning at extreme lengths. Empirically, transformers beat SSMs on tasks that require precise long-range associations (multi-document QA, complex reasoning across passages).

Mamba and Mamba-2: how the state actually works

The "no KV cache" claim deserves unpacking, because the underlying machinery is genuinely different from attention and shapes everything about how to think about long-context inference on these models.

The selective state-space mechanism. Mamba's core operation, applied at every layer, is a selective scan over a hidden state. The state has a fixed dimension (typically d_state = 16 or 64) per channel. As tokens stream in, the state is updated:

h_t = A_t * h_{t-1} + B_t * x_t      (state update)
y_t = C_t * h_t                      (output)

The matrices A_t, B_t, C_t are input-dependent — derived from the current token's hidden state. This is the "selective" part: Mamba can choose whether to remember or forget, conditioned on what it just saw. Pure linear attention or earlier SSMs (S4) had fixed A, B, C matrices independent of input — that's why they underperformed on language modeling.

The state size is fixed. For a Mamba-2 model with d_state=64, hidden_size=4096, the per-token state is 64 × 4096 × bytes_per_element. At BF16: 512 KB per layer per sequence. For a 32-layer model: 16 MB per sequence. That's it. Constant. Doesn't grow with context.

A 1M-token sequence on Mamba uses the same 16 MB of state as a 1-token sequence. The only thing that changes is computation time (linear in tokens), not memory.

Training in parallel, inference recurrently. Mamba's clever trick: the state-space recurrence has a parallel formulation that can be trained efficiently on GPUs (via the SSD parallel scan kernel in Mamba-2). At inference, you switch to the recurrent formulation, which is O(1) per token instead of O(N) for attention.

The asymmetry: training a Mamba model is roughly as expensive as training a transformer of similar parameters (both need parallel scans of length N). Inference is dramatically cheaper at long context. This is why Mamba models are increasingly attractive for deployment, even if training pipelines are mature for transformers.

Where Mamba breaks. Pure Mamba models have measurably weaker in-context learning than transformers. Specifically:

• Multi-document needle-in-haystack: Mamba ranks ~10–20% lower than a similarly-sized transformer.
• Complex multi-hop reasoning: Mamba struggles with tasks that require holding multiple distinct facts and combining them.
• Long-form code generation with cross-file dependencies: Mamba quality drops past ~50k tokens.

The intuition: Mamba's state is a fixed-size summary. It necessarily compresses information across all prior tokens. For tasks where you need to attend to specific past tokens with high fidelity, attention's full KV cache wins.

This is why pure Mamba hasn't displaced transformers, despite the inference economics. The quality gap on the tasks people actually care about is real.

Hybrid architectures

The current frontier: hybrids that get most of the SSM efficiency with most of the transformer quality.

Jamba (Lieber et al., 2024, AI21): every 8th layer is full attention; the rest are Mamba blocks. KV memory is ~1/8 of pure transformer; quality matches Llama-3 70B on most benchmarks.

Mamba-2 hybrids: similar pattern, slightly different ratios.

Goldfish (Snowflake AI Research): SWA + occasional full-attention layers.

If you are designing for million-token contexts on cheap hardware, the answer is probably hybrid, not the next round of KV quantization. The cost ceiling on dense attention is real and physics, not engineering.

Jamba layer pattern in detail

Jamba (Lieber et al., AI21, 2024) is the hybrid that matters most for production today, because AI21 ships it and the inference economics are visible. The architecture:

A 52B parameter Jamba model is built as a stack of "Jamba blocks". Each block contains 8 layers in a specific pattern:

Layer 1: Mamba (SSM)
Layer 2: Mamba
Layer 3: Mamba
Layer 4: Attention (full attention with GQA)  ← only attention layer in this block
Layer 5: Mamba
Layer 6: Mamba
Layer 7: Mamba
Layer 8: MoE (mixture of experts on the FFN)

So for every 8 layers of compute, only 1 has an attention KV cache. The other 7 use Mamba state.

Total per-token "memory state" for Jamba 52B at long context:

• Attention KV (1 layer per block × N blocks): tiny.
• Mamba state (7 layers per block × N blocks): fixed at ~few MB total per sequence.

For Jamba 52B compared to a transformer of equivalent quality (Llama-3 70B-class):

• Llama-3 70B at 256k context: 80 GB per request KV.
• Jamba at 256k context: ~10 GB per request (and most of that is the few attention layers).

The attention-layer KV is what gives Jamba its in-context learning quality. Pure Mamba couldn't compete on multi-document QA; Jamba can because every 8th layer is full attention with full KV. The Mamba layers handle the bulk of the linguistic processing efficiently; the attention layers handle the cross-document associations.

This is the architectural sweet spot for 2026: most of the SSM efficiency, most of the transformer quality, no need to push KV quantization to absurd levels. If you're building products that need >256k context, Jamba and similar hybrids should be on your shortlist.

What's the right layer ratio?

Jamba uses 1:7 (attention:SSM). Other hybrids vary:

• Zamba2 (Zyphra, 2024): 1:6 attention:Mamba ratio. Smaller models (2.7B), competitive quality.
• Hymba (NVIDIA, 2024): 1:1 — every layer combines attention and Mamba in parallel. Higher KV cost but better quality per parameter.
• Goldfish (Snowflake): SWA layers + occasional full-attention layers, no Mamba. KV is bounded by the SWA window.

The optimal ratio depends on workload. For long-context retrieval, more attention layers help. For pure language modeling perplexity, fewer attention layers are fine. For agentic reasoning, intermediate ratios seem to win.

Expect 2026–2027 to bring more empirical work on layer ratios, and probably a convergence around 1:4 to 1:8 attention:SSM for most workloads.

Deploying hybrids in 2026

The serving stack story for hybrids is still maturing:

• vLLM: Mamba support added in 2024 via custom kernels. Jamba support is partial as of mid-2025.
• SGLang: similar — community contributions for SSM support are recent.
• TRT-LLM: NVIDIA-internal SSM kernels exist; public TRT-LLM Mamba support is improving.
• MLX (Apple Silicon): Mamba kernels are well-tuned; Macs are surprisingly competitive for hybrid model inference at moderate scale.

For production deployments of Jamba or Mamba-2-based models in 2026: expect to use the model author's official inference code first (AI21 for Jamba, Mistral or NVIDIA for their hybrids) rather than vLLM/SGLang. The open-source ecosystem is catching up but still has rougher edges than for pure transformers.

By 2027, this should normalize. Hybrid models should be just as easy to serve as transformers, with the same paged-attention and prefix-caching benefits applied to the attention layers, and SSM-specific optimizations applied to the recurrence layers.

---

Capacity planning: three worked examples

Let's apply the math to three realistic scenarios.

Scenario 1: 8× H100 serving Llama-3 70B at 32k context

Setup: 8× H100 80GB. Llama-3 70B Instruct. Target 32k context, P95 first-token latency 2s, 10 concurrent active users at peak.

Step 1: model fit. BF16 weights = 140 GB. Doesn't fit on one GPU. TP=2 spreads it across 2 GPUs (70 GB each). With CUDA reserved + activations + NCCL buffers, that's ~74 GB used per GPU at idle, leaving ~6 GB free. Not enough for KV.

Move to FP8 weights: 70 GB total, 35 GB per GPU on TP=2. Now ~40 GB free per GPU for KV.

Step 2: KV math at TP=2. Per-GPU KV per token = 2 × 80 × 4 × 128 × 1 (FP8 KV) = 80 KB. At 32k context: 2.56 GB per request per GPU.

With 40 GB free: ~15 concurrent requests × 32k each fits before OOM. Use 12 (leave headroom for dynamic activation memory and for prefill bursts).

Step 3: replicate. 2 GPUs serve up to 12 concurrent. With 8 GPUs, run 4 replicas of TP=2. Total throughput: 4 × 12 = 48 concurrent active users at 32k.

Step 4: prefix caching. If your system prompt is 1k tokens shared across all users, prefix-caching effectively gives you ~1.4× capacity (depends on user mix). Boost: 48 → ~67 concurrent.

Step 5: validate latency. Llama-3 70B FP8 on 2× H100 prefills 32k tokens in ~1.4s on TRT-LLM with FP8 KV. First-token latency is dominated by this. P95 = 2s budget means: prefill time + queue wait < 2s. With 12 concurrent requests at 12 batched prefills/sec, queue wait is ~0 on average. Met.

Result: 8× H100 with TP=2 × 4-replica + FP8 weights + FP8 KV + paging + prefix caching can serve ~67 concurrent users at 32k context with P95 < 2s for first token.

Scenario 2: Single H200 serving DeepSeek-V2 at 128k context

Setup: 1× H200 141GB. DeepSeek-V2 (236B total, ~21B active per token, MLA architecture). Target 128k context.

Step 1: model fit. DeepSeek-V2 in FP8 weights ≈ 240 GB total. Doesn't fit on H200. Need at least TP=2 or model-pruning... wait, but we said single H200.

Actually, let's redo: DeepSeek-V2 Lite (16B, MLA) in FP8 ≈ 16 GB. Fits on one H200 with 125 GB to spare. Use this for the example.

Step 2: KV math. MLA gives ~10 KB per token (estimate; verify with DeepSeek's actual config). At 128k context: 1.28 GB per request. On 125 GB free, that's ~95 concurrent 128k-context requests.

Compare to GQA-8 model of similar capability: a Llama-3 8B in FP8 KV would be ~64 KB/token, 8.4 GB at 128k, ~14 concurrent requests in the same memory budget. MLA's advantage compounds.

Step 3: validate. This is the regime where MLA's economics shine. DeepSeek's API pricing reflects this: their long-context tokens are ~1/5 the cost of comparable competitors. The KV efficiency is the underlying reason.

Scenario 3: B200 serving frontier MoE at extreme context

Setup: 1× B200 192GB. Hypothetical 405B MoE with 56B active, GQA-8, BF16 weights would be ~810 GB — way too big. Skip this combination.

Instead: 1× B200 serving Llama-3.3 70B FP8 at 1M-token context.

Step 1: model fit. 70B FP8 = 70 GB. Leaves 122 GB on B200.

Step 2: KV math. 320 KB/token BF16 → 160 KB/token FP8. At 1M context: 160 GB per request. Doesn't fit.

Even on B200, 1M-token dense attention with full KV per request doesn't fit. You need:

• More GPUs (TP=4 across 4× B200 = 40 KB/token/GPU = 40 GB/request/GPU; fits 2-3 in-flight requests).
• Or hierarchical KV (NVMe offload of cold blocks).
• Or hybrid architecture (SWA + full attention pattern).

Conclusion: 1M-token serving requires architectural choices, not just bigger HBM. B200's 192 GB doesn't trivialize the problem; it just moves the threshold.

---

Cost economics: why position matters

Most hosted APIs charge per token regardless of context. That pricing is a leftover from the short-context era.

The provider's actual cost shape

The marginal cost to a provider of generating one more token isn't constant. It's roughly:

cost(token at position n) ≈ base + α × n

where α is set by KV-cache occupancy time × KV-cache size at that position. The position-dependent term grows linearly with n, not with log(n) or anything sub-linear, because:

1. The KV cache for a request at position n is n × kv_per_token bytes.
2. That cache occupies GPU memory for the duration of the request.
3. The GPU has finite memory; capacity for one long-context request is capacity not available for other requests.
4. The opportunity cost of capacity scales linearly with how much capacity is occupied.

Back-of-envelope numbers

For Llama-3 70B at $2/H100-hour, FP8 KV, GQA-8, with realistic batching and utilization assumptions:

• Token at position 4k: ~$0.40 / M output tokens (KV cost amortized).
• Token at position 32k: ~$1.20 / M output tokens (3× the KV burden).
• Token at position 100k: ~$3.50 / M output tokens (8× the KV burden, plus prefill amortization).
• Token at position 200k: ~$8 / M output tokens (16× the KV burden, queue effects start dominating).

These are rough — your provider's economics depend on hardware mix, utilization, batching efficiency. But the shape is universal: pricing per token at long context cannot stay flat forever.

Provider response in 2025–2026

Most frontier providers have already moved to tiered or quadratic-ish pricing past 32k:

• OpenAI introduced batched API with up to 50% discount but only for short-context tier.
• Anthropic Claude pricing tiers are flat per-context-tier but the gap between 8k and 200k tier prices is roughly the gap our model predicts.
• Google Gemini charges differently for ≤128k vs >128k input.
• DeepSeek offers long-context at ~1/5 the price of competitors, which is plausible only because of MLA's KV efficiency.

Watch this trend continue. By 2027, expect quadratic-ish pricing (or position-dependent metering) to be standard at the API level.

What this means for buyers

If you're building products on hosted LLM APIs:

• Don't assume per-token cost is constant across context positions. Model your unit economics with position-dependent pricing.
• Long contexts are not "just a bit more expensive"; they can be 5–10× more expensive at the position level.
• When a frontier provider offers a flat price across context lengths, treat it as a temporary pricing strategy, not a stable economic equilibrium.

If you're operating your own inference:

• Measure your effective cost per output token at different context positions. The math above gives you the order of magnitude; your actual numbers depend on hardware utilization.
• Consider context-tiered pricing at your own product layer if you serve external users.

---

Stack comparison: vLLM, SGLang, TRT-LLM, TGI, LMDeploy, llama.cpp

Six serving stacks dominate in 2026. Here's how they handle KV cache.

vLLM

Originated paged attention (Kwon et al., 2023). Now the most popular open-weight inference engine.

• Paging: yes, default. block_size=16 default, configurable.
• Prefix caching: yes, default since v0.6.0.
• KV quantization: FP8 e4m3, INT8, partial INT4 via integration.
• Offload: experimental CPU offload via --cpu-offload-gb.
• Speculative decoding: yes, EAGLE-2 default.
• TP/PP: full support.

Best for: general-purpose multi-tenant production deployments. The default safe choice.

SGLang

Built on top of paged attention, extends with RadixAttention.

• Paging: yes, default.
• Prefix caching: RadixAttention — the most aggressive sharing in any production stack.
• KV quantization: FP8 e4m3.
• Offload: not supported as of mid-2025.
• Speculative decoding: yes.
• TP/PP: full support.

Best for: chat-heavy workloads with shared system prompts, structured output, agentic workloads where prefix sharing dominates.

TensorRT-LLM (NVIDIA)

NVIDIA's first-party engine. Fastest on H100/H200/B200 hardware, but locked to NVIDIA.

• Paging: yes.
• Prefix caching: yes.
• KV quantization: FP8, INT8, FP4 (Blackwell only).
• Offload: CPU and NVMe offload supported.
• Speculative decoding: yes, EAGLE-2 and MEDUSA.
• TP/PP: full support, optimized.

Best for: single-tenant deployments on NVIDIA hardware where maximum throughput is the goal. The downside is build complexity (TRT-LLM requires building a model-specific engine, less plug-and-play than vLLM).

TGI (Hugging Face)

Hugging Face's serving stack, powering HF Inference Endpoints.

• Paging: yes.
• Prefix caching: partial.
• KV quantization: FP8.
• Offload: not supported.
• Speculative decoding: partial.
• TP/PP: TP supported.

Best for: deployments on HF Inference Endpoints, model deployments where HF integration matters more than peak performance.

LMDeploy

Chinese-developed stack, strong on Chinese open-weight models.

• Paging: yes.
• Prefix caching: yes.
• KV quantization: yes via --quant-policy 8.
• Offload: not supported.
• Speculative decoding: yes.
• TP/PP: TP supported.

Best for: Qwen, DeepSeek, ChatGLM and other Chinese open-weight models where LMDeploy has model-specific optimizations.

llama.cpp

CPU-first, GGUF format, runs on basically anything.

• Paging: yes (KV cache management, less aggressive than vLLM-style paging).
• Prefix caching: basic.
• KV quantization: q8_0, q4_0, etc. via cache type flags.
• Offload: yes — memory-mapped weights, runs models that don't fit in RAM by streaming from disk.
• Speculative decoding: not in core (community plugins exist).
• TP/PP: limited.

Best for: local single-user inference, weird hardware (Apple Silicon, AMD, CPU-only), edge devices.

Pick by workload

• Multi-tenant production, general-purpose: vLLM.
• Chat with shared system prompts: SGLang.
• Single-tenant max throughput on NVIDIA: TRT-LLM.
• HF ecosystem: TGI.
• Chinese open-weight models: LMDeploy.
• Local / single-user / weird hardware: llama.cpp.

If you're starting fresh and don't have a specific reason, start with vLLM. Switch later if you measure a specific bottleneck.

---

Comparative benchmarks

Numbers are the only honest way to compare serving stacks. The catch: benchmarks are workload-specific, and someone else's benchmark on someone else's traces is rarely directly applicable to your situation. What follows is a synthesis of public benchmarks (LMSys, vLLM blog, NVIDIA TRT-LLM benchmarks, SGLang's own measurements) circa 2025–2026, normalized as best as possible. Run your own — these are starting points.

Llama-3 70B on 1× H200 141 GB, FP8 weights, FP8 KV

Single-replica throughput on three workload archetypes:

Stack	Chat (1k system + 500 input + 200 output, no shared prefix)	Chat (shared 1k system)	RAG (8k input + 500 output)
vLLM 0.6+ (paged + prefix cache)	850 tok/s	2100 tok/s	540 tok/s
SGLang 0.4+ (RadixAttention)	880 tok/s	3200 tok/s	580 tok/s
TRT-LLM 0.13+	1100 tok/s	2400 tok/s	640 tok/s
LMDeploy 0.6+	920 tok/s	2200 tok/s	560 tok/s

Reads:

• TRT-LLM wins on raw single-stream throughput — its custom kernels and engine-build approach are optimized for one model, one GPU.
• SGLang wins decisively on shared-prefix workloads. RadixAttention is doing exactly what it's designed for here.
• vLLM is the consistent middle. Never the fastest, never the slowest. Most diverse model support.

Llama-3 70B on 4× H100 80 GB (TP=4), FP8 KV, 32k context

Stack	Concurrent users at 32k	Throughput (tok/s aggregate)	First-token P95 latency
vLLM 0.6+	24	1800	2.1 s
SGLang	28	2050	1.9 s
TRT-LLM	22	2200	1.6 s
LMDeploy	24	1850	2.2 s

The 4-GPU TP=4 setup is more forgiving — overall throughput is up across the board. Differences narrow.

DeepSeek-V2 (236B MoE, MLA) on 8× H100 80 GB (TP=8)

This is the "MLA pays off" benchmark. DeepSeek-V2 with MLA has ~70 KB/token KV vs ~320 KB/token for Llama-3 70B at similar capability.

Stack	Concurrent users at 128k	Throughput (tok/s aggregate)
vLLM (with MLA support)	18	1100
DeepSeek's own stack	24	1450
SGLang	16	950
TRT-LLM	n/a (MLA support pending)	n/a

DeepSeek's first-party stack handles MLA most efficiently. Open stacks were still adding MLA optimizations as of mid-2025; numbers above represent the state of integration in early 2026.

Throughput vs latency trade-off

Higher throughput often comes with higher tail latency. A specific measurement on Llama-3 70B + vLLM:

• max_num_seqs = 16: 1200 tok/s, P95 first-token 1.4s.
• max_num_seqs = 32: 1800 tok/s, P95 first-token 2.1s.
• max_num_seqs = 64: 2200 tok/s, P95 first-token 3.5s.
• max_num_seqs = 128: 2400 tok/s, P95 first-token 6.2s (queueing dominates).

There's no free lunch. If P95 latency matters (most user-facing chat does), don't push concurrency too high. If throughput matters more (batch workloads, agentic workloads where step latency is tolerable), push it higher.

Benchmarks to ignore

A few common benchmark patterns produce misleading numbers:

• Single-stream throughput on a single short prompt. Doesn't exercise paging, prefix caching, or batching. Looks great in marketing materials, irrelevant in production.
• Aggregate throughput at saturation on uniform workload. Highly stack-dependent, but rarely matches mixed real-world traffic.
• "Time to first token" without context-length specification. Prefill scales quadratically with context; an unspecified TTFT number is meaningless.

Always insist on: workload distribution (context lengths, batch shape, prefix overlap), hardware spec (GPU, count, TP/PP), KV format, and which percentile of latency the throughput number targets. If a vendor benchmark omits any of these, treat it as marketing.

Run your own

The right way to evaluate stacks is to capture 10–60 minutes of your actual production traffic (anonymized), replay it against each candidate stack, and measure throughput, latency, and resource usage. vLLM and SGLang both ship benchmark scripts that take a recorded trace and replay it.

# vLLM's benchmark_serving.py replays a JSONL trace
python benchmarks/benchmark_serving.py \
  --backend vllm \
  --base-url http://localhost:8000 \
  --dataset-path your-trace.jsonl \
  --num-prompts 1000

A two-day investment in trace-driven benchmarking saves months of debugging mismatched performance later.

---

Migration guide

You have a working serving deployment. You want to adopt the optimizations in this guide. What's the order, what can break, how do you validate?

From contiguous to paged KV

If you're on vLLM 0.2+ or any stack from 2024 onward, you're already paged. This migration is for legacy deployments only.

What changes: KV memory layout, eviction behavior, attention kernels.

What can break: nothing user-visible. Output is bit-identical (modulo floating-point reordering). Throughput jumps 2–4×.

Validation: compare output on a fixed seed across before/after. Should match within FP rounding. Compare KV utilization metrics — should jump from ~40% to ~94%.

Risks: very low. This is the safest migration in the guide.

From FP16 KV to FP8 KV

What changes: KV cache size halves. Attention kernels do FP8 reads instead of FP16.

What can break:

• Quality drops 0.1–0.5 points on standard benchmarks if calibration is good. Larger drops if calibration is missing or wrong.
• Long-context retrieval (RULER) is more sensitive than short-context (MMLU). Test at your workload's context length.

Validation steps:

1. Run your eval suite at 4k, 16k, 64k context against BF16 KV baseline.
2. Switch to FP8 with --calculate-kv-scales. Re-run.
3. Compare. Quality should be within 1 point on every metric. If larger, your calibration set isn't representative — pick better calibration data.
4. Run a soak test for 24 hours. Watch for NaN propagation symptoms (gibberish output mid-generation).

Risks: moderate. The most common failure is missing calibration. Don't skip --calculate-kv-scales.

From no prefix caching to prefix caching

What changes: KV blocks for shared prefixes deduplicate. Throughput on workloads with prefix overlap jumps 2–10×.

What can break:

• Extremely rare correctness bugs (cached blocks getting mismatched with the wrong tokenizer state).
• Cache invalidation issues on model update.

Validation steps:

1. Enable --enable-prefix-caching.
2. Send the same request twice. Verify identical output.
3. Send a request, modify one token in the middle of the prefix, send again. Verify output reflects the modification (cache should invalidate at the divergence).
4. Update the model. Verify the cache is cleared automatically.
5. Soak for 24 hours.

Risks: low if your stack handles invalidation correctly. Moderate if you do anything custom with the cache.

From vLLM to SGLang for chat workloads

What changes: Replace the engine. Same model, same hardware, different KV management.

What can break:

• API differences. SGLang's HTTP API is similar but not identical to vLLM's. Client-side updates needed.
• Some advanced features (multi-LoRA, certain quantization formats) have different support levels. Verify before migrating.

Validation steps:

1. Stand up SGLang in parallel with vLLM. Same model, same hardware spec.
2. Replay a recorded trace against both. Compare throughput, latency, and output.
3. Specifically: measure prefix cache hit rate on SGLang. If it's not >50% on your workload, you're not getting the SGLang advantage and should stay on vLLM.
4. Cut over a small percentage of traffic. Watch error rates.
5. If clean, ramp to 100%.

Risks: moderate. Most teams who migrate to SGLang stay there. Some go back when their workload turns out not to have the prefix sharing they assumed.

Adding speculative decoding (EAGLE-2)

What changes: A draft model runs alongside the target. KV cache memory grows by ~10–15%. Throughput on agentic workloads jumps 2–3×.

What can break:

• Memory budget. Your existing KV-bound replicas may OOM after enabling spec-decode unless you give them more headroom.
• Quality. EAGLE-2 is exact (target model verifies, not the draft) but a bug in implementation could drift output. Validate.

Validation steps:

1. Run the EAGLE-2 setup procedure for your stack (specific draft model checkpoint, target model checkpoint, integration flags).
2. Reduce max_num_seqs by 10–15% to leave room for draft KV.
3. Verify output is identical to non-spec on a test set (it should be — spec-decode is exact).
4. Measure throughput improvement on your actual workload. If it's <1.5×, you may not be benefiting (workloads with high entropy decode less reliably).
5. Soak.

Risks: moderate. Memory budgeting is the most common stumble.

Adding hierarchical KV / NVMe offload

What changes: Cold KV blocks spill to local NVMe. Capacity for cold-cache workloads (very long context, low cache reuse) goes up 5–10×.

What can break:

• P95/P99 latency. Swap-in from NVMe takes 1+ seconds for large blocks.
• Local SSD wear (NVMe writes have lifetime limits, though this is rarely the bottleneck for typical inference write patterns).

Validation steps:

1. Configure NVMe offload (--enable-nvme-offload on TRT-LLM, equivalents on other stacks).
2. Run synthetic workload that exceeds GPU KV capacity but fits in NVMe.
3. Verify output is correct.
4. Measure latency tails. If P99 is acceptable, you're done.

Risks: high for interactive workloads (latency tails). Low for batch/analytical.

Order of migrations

If you're starting from a baseline 2023-era setup and want to reach 2026-state-of-the-art, do them in this order:

1. Update to a paged stack (vLLM, SGLang, TRT-LLM). Free 2–4× throughput.
2. Enable FP8 KV. Free 2× memory capacity.
3. Enable prefix caching. Free 2–10× throughput on shared-prefix workloads.
4. Add speculative decoding. 2–3× throughput on agentic workloads.
5. (Optional) Switch to SGLang if your workload has heavy prefix sharing.
6. (Optional) Add NVMe offload if you serve very long contexts.

Skip any step that doesn't apply to your workload. The order matters because each later step assumes the earlier ones are in place.

---

Production observability

Four metrics that tell you the truth about whether your KV strategy is working.

KV cache utilization

kv_utilization = used_kv_blocks / total_kv_blocks

What it tells you:

• Below 50%: over-provisioned. Either your max-batch is too low, your max-context is too low, or your block size is too small.
• 50–85%: healthy.
• Above 90% sustained: you're about to hit eviction. Add capacity or reduce admission rate.

Where to find it: vLLM exposes vllm:gpu_cache_usage_perc on /metrics. SGLang exposes kv_cache_utilization via the admin API.

Eviction rate

Preemption events per second.

What it tells you:

• Zero: headroom available.
• 0–5/sec: occasional preemption, probably fine.
• >5/sec sustained: you are throughput-bound on KV. Either reduce max-context, add replicas, or switch to a smaller KV format.

Where to find it: vLLM exposes vllm:preemption_total (counter). Take its derivative.

Prefix cache hit rate

For stacks that expose it.

What it tells you:

• <30%: prefix sharing isn't paying off. Consider a stack switch (vLLM → SGLang) or workload reshape (consolidate system prompts).
• 30–70%: typical for mixed workloads. Acceptable.
• >70%: you're getting a lot of free throughput. Protect this — avoid client-side prompt randomization, ensure tokenizer stability across deploys.

Where to find it: vLLM exposes vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total. Hit rate = hits/queries.

First-token latency P50/P95

Directly correlates to prefill time. Track per-context-bucket so you don't average away the long-context tail.

What it tells you:

• Per-bucket, you can see if a specific context length is causing problems (e.g., 64k+ requests are queueing).
• The ratio P95/P50 reveals tail behavior. Anything above 3× indicates lumpy load patterns.

Where to find it: vLLM vllm:time_to_first_token_seconds histogram. Bucket by request_input_tokens if you can.

Other metrics worth tracking

• Tokens-per-second (decode throughput): direct revenue indicator.
• Tokens-in-flight (active KV positions): correlates to GPU utilization.
• Queue depth: requests waiting for a free slot. Should be near zero in healthy operation.
• GPU memory free: should track inversely with kv_utilization.

vLLM exposes all of these via Prometheus on /metrics. SGLang exposes them via the admin API. If your stack doesn't expose at least the four above, you are flying blind.

---

Failure modes and troubleshooting

The bugs and operational gotchas that cost real money.

NaN propagation in FP8 KV

A single overflow during attention writes a NaN into the cache. Subsequent reads pollute the entire sequence. Sometimes the entire batch.

Symptom: model output becomes garbage tokens partway through generation. The model might output gibberish, repeat tokens, or output the special token for end-of-sequence.

Cause: usually missing or incorrect KV scales. Without proper per-channel calibration, FP8 can't represent the activation range accurately, and outliers overflow.

Fix: enable --calculate-kv-scales properly. If you're already calibrated and still seeing NaNs, fall back to FP16/BF16 KV until you debug. Verify your calibration data is representative of production traffic.

Block table corruption under heavy preemption

Race condition between eviction and incoming requests. Vanishingly rare, but spectacular when it happens.

Symptom: random tokens swapped between sequences. User A sees output mid-stream that looks like a response to User B's query.

Cause: a synchronization bug in the block manager during high-preemption regimes. Most prevalent on older vLLM versions (pre-0.6).

Fix: upgrade to a current stack version. If you're seeing this on a current version, file a bug — these are typically taken seriously.

Prefix cache invalidation on tokenizer changes

You update the tokenizer, redeploy, but don't clear the prefix cache. Cached blocks reference token IDs from the old tokenizer.

Symptom: corrupted output for any user whose prefix happens to land on a stale cached block.

Fix: clear the prefix cache on every model deploy. vLLM clears automatically when the model changes, but may not if only the tokenizer changes. Check your stack's behavior. When in doubt, restart.

TP rank desync on KV

When rank 0 evicts a block but rank 1 doesn't (rare, version-specific).

Symptom: hangs, asymmetric outputs across ranks, or NCCL collective failures.

Fix: upgrade your stack. This class of bug is mostly historical but worth checking changelogs before pinning a version. If reproducible, file a bug with the trace.

OOM during prefill of a single oversized request

A user sends 200k tokens; you allocated for 128k max. If you didn't set --max-model-len correctly, the request takes the whole replica down.

Symptom: replica crash, all in-flight requests drop, container restart.

Fix:

• Always set --max-model-len to a value you've actually capacity-planned for.
• Configure rejection at the API layer (return HTTP 413 for over-limit requests).
• Don't rely on the inference engine to gracefully reject — it should, but in practice it sometimes OOMs.

Inconsistent prefix sharing across replicas

Two replicas of the same model serving the same workload have very different prefix cache hit rates.

Symptom: replica A is at 85% hit rate, replica B is at 25%. Throughput is asymmetric.

Cause: load balancer is round-robining requests instead of hashing on prefix. Each replica builds its own cache independently and hits compound only when the same prefix lands on the same replica multiple times.

Fix: switch to consistent-hashing or session-affinity load balancing if your prefix patterns are stable enough to benefit. Or accept the asymmetry — if it's small, it's fine.

Slow swap-in causing P99 spikes

You enabled CPU offload with PCIe Gen4 hardware. Most requests are fine, but occasional swap-in adds 200ms+ to first-token latency.

Symptom: P95 latency is healthy, P99 is terrible.

Fix:

• Upgrade to PCIe Gen5 if possible.
• Reduce --cpu-offload-gb so fewer sequences are eligible for offload.
• Monitor swap-in events specifically and rate-limit if they exceed a threshold.

Debugging KV memory leaks

Memory usage grows over time, no apparent cause.

Symptom: replica works for hours then OOMs. KV cache shows healthy utilization throughout.

Possible causes:

• A bug in eviction where blocks aren't actually freed.
• Phantom requests stuck in the request manager.
• Activation memory leaking (rare; CUDA caching allocator usually handles this).

Debugging:

• Restart and watch memory growth pattern. Linear in time? Linear in requests served? Step changes at specific events?
• Use NVIDIA's nsys and nvprof to profile memory allocations.
• Check stack issue trackers — known leaks are usually fixed quickly in current versions.

---

Frequently asked questions

Q: Should I use FP16 or BF16 for the KV cache?

A: BF16 if your hardware supports it (Hopper, Ampere, Blackwell). BF16 has the same bit width as FP16 but a wider dynamic range, which is more forgiving for the activation magnitudes that KV stores. FP16 works fine in practice but has slightly more numerical edge cases (rare overflow on attention dot products with very-long-context retrieval).

If you're on hardware without BF16 support (e.g., older consumer GPUs), use FP16. The quality difference is negligible in most scenarios.

Q: Why is my GPU memory full but KV utilization shows 30%?

A: Several possible causes:

• The block size is too large for your typical sequence length, causing tail-block waste.
• Your gpu_memory_utilization setting reserved a smaller-than-needed fraction for KV.
• Activation memory or NCCL buffers are taking unexpected space.
• A memory leak (see Failure modes).

Profile with nvidia-smi and your stack's /metrics endpoint. If kv_utilization * gpu_memory_utilization adds up to ~70%, that's the answer (the other 30% is the model + activations + overhead). If it doesn't add up, something's leaking.

Q: Can I use INT4 KV cache in production?

A: Yes, with caveats:

• Workload must not be long-context retrieval-heavy. INT4 breaks RULER at 64k+.
• Use a stack that supports per-channel-K, per-token-V calibration (KIVI-style). Naive INT4 is much worse.
• Test on a representative workload before deploying. The quality cost is workload-dependent and you need actual numbers.

For most chat and code workloads, INT4 KV is fine and unlocks significant capacity. For RAG, document analysis, or anything that needs precise long-range retrieval, stick with FP8.

Q: What's the difference between paging and prefix caching?

A: Paging is the memory layout: the KV cache is divided into fixed-size blocks instead of contiguous per-sequence buffers. This eliminates fragmentation.

Prefix caching is the deduplication: when two sequences have the same prefix tokens, they share the same blocks instead of computing them twice. This requires paging (you need block-level granularity to share blocks).

Most modern stacks have both. Paging is invisible (it just makes you faster); prefix caching may need explicit enablement (--enable-prefix-caching on vLLM).

Q: Does GQA hurt model quality?

A: Marginally. On standard benchmarks (MMLU, HellaSwag, etc.), GQA-8 is within 0.5 percentage points of MHA. On long-context retrieval (RULER, needle-in-haystack), within 1–2 points.

The economic benefit (8× less KV memory) is enormous and the quality cost is small. Every modern open-weight model uses GQA for this reason. If you're training a model, use GQA-8 unless you have a specific reason not to.

Q: How do I know if I'm KV-bound or compute-bound?

A: Profile your serving:

• KV-bound: Adding more KV memory (smaller models, larger GPUs, FP8 KV) increases throughput. Adding more compute (faster GPUs, more parallelism) doesn't.
• Compute-bound: Adding faster compute increases throughput. KV-related changes don't.

Most production deployments at long context are KV-bound. Most at short context are compute-bound. You can test by changing one variable at a time and measuring.

Q: Should I run TP=2 or TP=4 for Llama-3 70B?

A: Depends on your context length and concurrent users.

• TP=2 halves per-GPU KV. Good for moderate context (≤32k) and moderate concurrency.
• TP=4 quarters per-GPU KV. Good for long context (≥64k) or high concurrency. The cost is more inter-GPU communication overhead.

A common pattern: TP=2 with FP8 KV at 32k is sweet for most production. TP=4 starts winning at 128k+ context or 100+ concurrent users.

Q: What happens to the KV cache when a request finishes?

A: Its blocks are freed and immediately reusable for new requests. There's no garbage-collection delay — the block manager moves them to the free list synchronously.

If prefix caching is enabled, the blocks may be retained in the prefix cache (waiting for a future request that shares the prefix) rather than freed immediately. Eviction from the prefix cache happens via LRU when capacity is needed.

Q: How does the KV cache interact with quantization-aware training?

A: The KV cache stores activations. Quantization-aware training (QAT) trains the model to be robust to weight quantization, but the activations (and therefore KV) are usually still trained in BF16 or FP16.

Some advanced QAT approaches train with FP8 activations to make the model robust to FP8 KV quantization at inference. This is research-level (not standard in 2026 open-weight training pipelines). For most practical purposes: train in BF16, deploy with FP8 KV, accept the small quality cost.

Q: Does prefix caching work across different versions of the model?

A: No. Prefix caching is keyed by token IDs and validity is contingent on the model weights being unchanged. When you redeploy with new weights, you must invalidate the prefix cache. If you don't, cached blocks contain KV computed with the old weights, and the model produces wrong output.

Most stacks invalidate the prefix cache on model reload automatically. Some don't if you only swap the LoRA adapter or some peripheral component. Always test after redeploy.

Q: What's the maximum useful context length on commodity hardware in 2026?

A: Depends on architecture:

• Dense transformer + GQA-8 + FP8 KV: ~256k–512k tokens on a single H200 with single-replica serving. Beyond that, you need multi-GPU or hierarchical KV.
• MLA-based models (DeepSeek-V2/V3): ~1M tokens on a single H200 due to ~5× smaller per-token KV.
• Hybrid (Jamba) or SWA models: ~5M+ tokens are physically possible because most layers don't grow KV with context. Quality at extreme lengths is the open question, not memory.
• Pure SSM (Mamba-2): arbitrarily long context with constant memory. Quality at extreme lengths is again the question.

For most production purposes, plan for 32k–128k as the standard, 256k as advanced, beyond as architectural-choice territory.

Q: How is KV cache size affected by attention sinks (StreamingLLM)?

A: Attention sinks (Xiao et al., 2024) keep the first 4 tokens plus the last N tokens in the cache, dropping everything in between for streaming inference.

Effect: the KV cache size becomes constant (4 + N tokens) regardless of how many tokens have been generated. This is a hard cap, useful for long streaming chats where you don't need long-range attention.

Trade-off: any task requiring attention to the dropped middle tokens fails. Retrieval over the past 100k tokens is impossible if you've dropped tokens 4 through 99996. Use this only for chat where local context dominates.

Q: Can I dynamically change the KV cache size during a request?

A: Generally, no. The KV cache for a request grows monotonically as tokens are generated. You can't "shrink" it mid-request without losing the dropped positions' attention.

Some experimental techniques (KIVI's online quantization, MiKV's compression) effectively do this by switching the cache to a more compact format mid-request. These are not standard.

Q: What's the difference between max_model_len and max_num_seqs?

A:

• max_model_len: the maximum context length per individual request. A single request can use up to this many tokens.
• max_num_seqs: the maximum number of concurrent sequences in flight at once.

The product max_model_len × max_num_seqs × kv_per_token should be ≤ your available KV memory. Most stacks enforce this implicitly by paging and rejection.

Q: Why does my prefill time scale quadratically with context?

A: Because attention is O(N²) over context. Prefilling 32k tokens does ~1024× more attention work than prefilling 1k tokens.

This is a fundamental cost of dense attention, not a KV-cache issue. Optimizations like FlashAttention reduce the constant factor (memory access, kernel efficiency) but don't change the asymptotic complexity. For very-long-context prefill, only architectural changes (sparse attention, SWA, SSMs) actually break the quadratic.

Q: Is the KV cache shared across user sessions?

A: Without prefix caching, no — each request has its own KV cache that's freed when the request finishes.

With prefix caching, yes — KV blocks for shared prefixes are deduped and persist in the cache until evicted. Users with overlapping prompts share the underlying KV storage transparently.

Q: How does KV cache work with batched prefill?

A: All requests in a batch are prefilled together: their tokens are concatenated, attention is computed for the combined sequence with proper masking to keep them isolated, and KV is written into per-request cache slots.

The throughput benefit: GPU utilization is higher because the batched matrix multiply is efficient. The latency cost: each individual request waits for the batch to be assembled.

Most stacks use continuous batching: instead of waiting for a fixed-size batch, they merge new requests into the in-flight batch as they arrive. This is the right default and is now standard.

Q: How does LoRA serving interact with the KV cache?

A: LoRA adapters modify the weight matrices, not the KV cache mechanism. When you serve multiple LoRA adapters on the same base model:

• The base model's KV cache works exactly as described in this guide.
• LoRA-specific computations happen on top of the base model's attention output. The K and V values cached are based on the base model's projection matrices, not the LoRA-adapted ones, in most stacks.
• This means swapping LoRAs mid-session usually doesn't invalidate the KV cache — but verify with your specific stack version. Some stacks bake LoRA into the K/V projections, in which case adapter switches do invalidate.

Multi-LoRA serving (S-LoRA, Punica, vLLM's multi-LoRA mode) typically caches base-model KV and applies LoRA adjustments at compute time. Memory overhead per LoRA is small.

Q: What about multi-modal models — how do image/video tokens affect KV?

A: Multi-modal models (Llava, Pixtral, Llama-3.2-Vision, Qwen2-VL) typically encode images/video into a sequence of tokens that are concatenated with text tokens before attention. From the KV cache's perspective:

• Image tokens are just tokens. They occupy KV cache slots like text tokens.
• Image encoding is dense — a single image often contributes 100–1000+ tokens depending on resolution and patch size.
• Video encoding can produce thousands of tokens per second.

This means a multi-modal request with one 1024×1024 image at typical patch sizes contributes ~256–1024 tokens just for the image, in addition to the text. Long video can dominate the context budget. Plan KV capacity for the multi-modal token cost, not just the text token count.

Some recent multi-modal architectures (e.g., NVIDIA's Sana, Apple's MM-Ferret) use specialized encoders that emit fewer tokens. Read the model's docs for the actual token-per-image count before sizing.

Q: How does the KV cache interact with reasoning models like o1?

A: Reasoning models (OpenAI o1, DeepSeek R1, Claude with extended thinking) generate long internal "thinking" sequences before producing the user-visible answer. From the KV cache's perspective:

• The thinking sequence is just generated tokens. KV grows as it generates.
• Thinking can be 1k–10k+ tokens for a single user query. Output context is much longer than input.
• This shifts the KV cost profile: traditional chat is "long input, short output" (input dominates KV). Reasoning is "moderate input, very long output" (output dominates KV).

Sizing implications: if you serve reasoning models, your effective context is input + thinking + output, not just input + output. Your max-context-len setting needs to accommodate the thinking budget.

Some providers separate thinking-mode KV from chat KV cache pools (different replicas, different settings). This is reasonable when usage patterns are very different.

Q: Does the KV cache help with quantization-aware fine-tuning?

A: Not directly. The KV cache is an inference-time concept. Quantization-aware fine-tuning happens at training time, when the cache isn't really a cache — it's just the activation tensor used for backprop.

The connection: training in quantized precision (FP8 or INT8 forward pass) makes the model's activations more robust to quantization at inference. So a model fine-tuned with FP8 forward will produce KV values that are well-suited to FP8 KV at deployment. This is "quantization-aware training" in the loose sense, and it's emerging in 2025–2026 frontier training pipelines.

For most production work: don't overthink this. Train as normal, deploy with FP8 KV, accept the small quality cost.

Q: Can I share KV cache across multiple requests for the same prompt?

A: Yes, that's exactly what prefix caching does. Two requests with the same prompt prefix share the same KV blocks. See the Prefix caching section.

If you're asking about something more aggressive — e.g., two completely different requests where you want to manually share KV — you can't. The KV cache is positionally indexed; tokens at position 100 of request A are not interchangeable with tokens at position 100 of request B unless the prefixes match exactly.

The exception is some research approaches (e.g., shared semantic KV) that look promising in 2025 but aren't yet production-grade.

Q: What's the difference between chunked prefill and continuous batching?

A: They're orthogonal optimizations:

• Continuous batching (Yu et al., Orca, 2022): instead of waiting for a fixed batch, dynamically merge new requests into the in-flight batch as they arrive. This is at the request level.

• Chunked prefill (vLLM 0.6+): split a single request's prefill into chunks instead of one big prefill. Useful for very long inputs where one prefill would block decode for too long. This is at the token level within a request.

Modern stacks use both. Chunked prefill is especially useful when you have mixed long-context and short-context requests; without it, a 64k-token prefill blocks decode for 1+ seconds, hurting P95 for everyone.

Q: How is KV cache different on AMD MI300/MI350 GPUs?

A: Mostly the same. The math (2 × layers × kv_heads × head_dim × bytes) doesn't care about hardware. The KV management strategies (paging, prefix caching, eviction) work the same.

Differences:

• AMD's HBM bandwidth and capacity differ from NVIDIA's (MI300X has 192 GB / 5.3 TB/s; MI355X has 256 GB). Per-GPU KV capacity is generally larger; bandwidth is comparable to H100.
• Quantization formats supported differ. FP8 is supported on MI300+; FP4 is upcoming on MI400+.
• Stack support varies. vLLM has AMD support; SGLang's AMD support is less mature; TRT-LLM is NVIDIA-only.

For Qwen and other open-weight models, AMD MI300X is a viable alternative. Verify your specific stack's AMD path; it's mostly there but with some bumps as of early 2026.

Q: What about Apple Silicon? Can I cache KV efficiently on M-series Macs?

A: Yes, with caveats. Apple Silicon's unified memory means CPU and GPU share the same memory pool — there's no PCIe transfer for KV. Latency-wise this is great.

Caveats:

• Total memory is the constraint. M3 Max has 128 GB max; M4 Ultra has 192 GB max. KV cache competes with weights and OS for that pool.
• llama.cpp and MLX are the primary serving stacks. Both support paged-style KV management, but kernels are less optimized than CUDA paths.
• Long context on Apple Silicon is feasible for single-user serving (your laptop, your inference). Not competitive for multi-user production at scale.

Sweet spot: local development, demos, edge deployments. Not a substitute for H100/H200 for production multi-tenant serving.

Q: How do I size KV cache for a chat that grows over many turns?

A: Multi-turn chat is one of the trickiest KV sizing problems because the request's effective context grows with every turn.

The math: turn 1's KV is tokens_in_message_1. Turn 2's KV is tokens_in_message_1 + tokens_in_response_1 + tokens_in_message_2. After 10 turns, you might have 10k+ tokens of context per session.

Sizing strategies:

• Bound the conversation. Cap turns at N or context at K tokens. Drop or summarize history beyond.
• Use prefix caching. Multi-turn chat has 100% prefix overlap with the prior turn (the new turn's prefix is the prior turn's full context). Prefix caching means turn N reuses turn N-1's KV.
• Plan for the long tail. Some users have very long chats. Decide: do you reject (HTTP 413), summarize (lose detail but bound cost), or accept (uncapped KV burden)?

OpenAI's ChatGPT, Claude, and Gemini all use combinations of these. The provider-side decision is invisible to the user but heavily affects your cost model if you build on these APIs.

Q: Can I serve different models from different KV cache pools on the same GPU?

A: With the right serving stack: yes. vLLM and SGLang support multi-model serving (multiple base models on one inference server) with separate KV pools per model.

Caveats:

• Memory must be partitioned. The pools don't share — if model A's pool fills, you can't borrow from model B.
• Most production deployments do this with separate replicas (one model per process), not separate pools in one process. The latter has marginal benefits and adds complexity.

Multi-LoRA on one base model is different — that's one KV pool shared across LoRA adapters. Multi-model is multiple base models with multiple pools.

Q: Why does my chat-with-RAG application have such low prefix cache hit rates?

A: RAG retrieves different passages per query, so each query has a different prefix even if the system prompt is shared. The system prompt blocks share; the retrieved-passage blocks don't.

To improve:

• Ensure deterministic retrieval. If the same query retrieves the same passages, repeat queries cache.
• Re-rank to a stable top-K. If retrieval scoring is noisy, two near-identical queries might get different passages and miss the cache.
• Cache strategies above the LLM. Some teams cache the entire RAG-retrieved-context-plus-LLM-response at the application layer, hitting the LLM only when the question is novel.

If your prefix cache hit rate on RAG is below 30% even with these in place, it's a property of your workload, not a fixable problem. Move on to other optimizations.

Q: Should I worry about KV cache security?

A: Yes, briefly. Multi-tenant inference services with prefix caching can leak information across tenants if not careful:

• Side-channel via timing: a request that hits the prefix cache responds faster than one that misses. If user A and user B share an unusual prefix, their cache hit/miss timing reveals that they're using related prompts.
• Tenant isolation: most production multi-tenant services scope prefix caches per tenant or per session. Cross-tenant sharing is usually disabled by default. Verify your stack's settings.
• Prompt injection: if cached prefix blocks contain attacker-controlled content (e.g., from a system prompt that includes user-supplied data), all downstream requests with that prefix are affected. Sanitize what you cache.

For most deployments, the defaults are sane. If you serve a multi-tenant production service with strict isolation requirements (compliance, regulated industries), audit the cache scoping configuration explicitly.

Q: How does FP8 KV interact with FlashAttention-3?

A: FlashAttention-3 (Tri Dao, 2024) has native FP8 KV support on H100/H200. The K and V tiles are loaded in FP8, dequantized to FP16 in shared memory at the boundary, the attention matmul runs in FP16, and the output is FP16. The HBM read bandwidth halves vs BF16 KV; the compute is unchanged. On B200, FA-3 supports FP8 KV plus FP8 attention compute via Blackwell's FP8 matmul, giving another ~1.5x speedup at the kernel level. The kernel is the reason FP8 KV "just works" without quality loss — quantization happens at memory boundaries, not inside compute.

Q: How is the KV cache structured for ring attention / context parallelism?

A: Ring attention shards the sequence across GPUs and rotates K/V tiles through a ring of devices. Each GPU holds 1/N of the KV at any moment; over the course of one attention computation, every GPU sees every K/V tile by rotating them. This makes context lengths beyond what fits on a single GPU possible at the cost of inter-GPU bandwidth proportional to (sequence_length × hidden_dim) per layer. Standard for 1M+ context serving in 2026. See the long-context attention guide.

Q: Why is prefill compute-bound when decode is memory-bound?

A: Prefill processes the entire prompt in one forward pass — sequence_length tokens per layer, all in parallel. The arithmetic intensity is high (compute per byte loaded scales with seq_len), so it saturates tensor cores. Decode processes one token per step per layer; the same model weights are re-loaded for each token, with very low compute per byte loaded. The transition point is at batch_size × seq_len ~ 230 for H100 (the arithmetic intensity needed to saturate tensor cores). Most chat decode batches sit at 10-50 effective parallel sequences, well below the compute-bound regime.

Q: How do I compute the KV cache size for a multi-modal model (Llava, GPT-4V)?

A: Vision tokens are concatenated with text tokens in the attention sequence. For Llava-1.5 13B with CLIP-ViT-L: each image is tokenized to 576 vision tokens that occupy KV positions just like text tokens. A 1024-text-token + 1-image request has effective seq_len of 1600. For video, multiply by frame count: a 30-frame video at 576 tokens/frame = 17280 KV positions for the video alone. This is why long-form video LLMs (LongVA, Video-LLaVA) need either MLA-style compression or aggressive frame downsampling.

Q: Can FP4 KV cache work in production yet?

A: Mid-2026, only on Blackwell (B200/GB200) and only in research-grade implementations. The TRT-LLM B200 path supports FP4 weights and FP8 KV; full FP4 KV is in development. The quality risk is high — FP4 KV drops more meaningful precision than FP8 KV, with measurable perplexity regression on most evals. Most production teams using B200 today run FP8 KV, not FP4. Expect FP4 KV to become viable in 2026-Q4 or 2027 as calibration techniques mature. See quantization tradeoffs for the broader FP4 picture.

Q: How does the KV cache interact with disaggregated prefill/decode?

A: The KV is computed on the prefill worker and transferred to the decode worker over NVLink/InfiniBand/RoCE. Transfer size: 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element. For Llama-3 70B at 8k context FP8 KV: 1.3 GB per request. Over a 400 GbE link, that's ~26 ms transfer time. Over NVLink 4 (900 GB/s): ~1.4 ms. The disaggregated topology relies on this transfer being fast relative to prefill latency. See disaggregated inference and InfiniBand vs RoCE.

Q: How does KV cache work on TPU v5/v6?

A: TPUs use HBM similar to GPUs but with different sharding patterns. JAX/Flax serving on TPU pods uses model parallelism across the pod's torus topology; the KV cache is sharded along the head dimension across chips. There's no PagedAttention equivalent in production on TPU — Google's Gemini serving uses bespoke memory management that's not publicly documented. The capacity math is the same; the engineering practices are different.

Q: How do I monitor KV cache fragmentation in production?

A: Key metrics: (1) kv_cache_usage (fraction of pool allocated), (2) kv_cache_fragmentation (allocated minus actually-used positions), (3) num_preempted_requests (count of requests evicted due to KV pressure), (4) block_pool_utilization (paged-attention block fill rate). vLLM exposes all four via /metrics Prometheus endpoint. Healthy production: fragmentation < 15%, preemption rate < 1%, block pool utilization 80-95%. Above 95% utilization, latency tails inflate as the pool thrashes.

Q: What is the right approach for KV-cache compression beyond FP8?

A: Three techniques in 2026 production use: (1) KIVI (per-channel INT4 for K, per-token INT4 for V) — published by Liu et al. 2024, supported in vLLM 0.7+; quality loss is workload-dependent but typically within 1 point of perplexity; (2) H2O (heavy-hitter oracle) — keeps only the top-K most-attended-to tokens in KV, drops the rest; works on long-context but loses information beyond a threshold; (3) StreamingLLM — keeps attention sinks plus a sliding window; useful for unbounded chat. None of these match FP8 KV's clean "drop-in no quality loss" profile, but they're necessary at very long context where FP8 alone isn't enough.

Q: Does the KV cache need to be on the GPU, or can it live in CPU RAM during idle time?

A: For active sequences in the decode loop, the KV must be on the GPU — attention reads it every step. For idle sequences (paused conversations, long-running agent sessions), CPU offload via PCIe is supported in vLLM and SGLang. The reactivation cost is ~50-200 ms for typical session sizes (transfer back to GPU); for very long sessions (1M context), reactivation can take 5-10 seconds. NVMe-tier offload (cold storage) is supported but practically used only for batch and offline scenarios — see the offloading section.

Q: How do I troubleshoot OOM on KV cache when starting a new request?

A: Common causes, in order of frequency: (1) max_model_len too high — request reserves max-context KV upfront; lower the limit or enable chunked prefill; (2) fragmentation — old requests left gaps in the pool; restart the worker or wait for cleanup; (3) block size too large — 32-token blocks fragment more than 16-token blocks at the cost of slightly more metadata; (4) model + KV approaching HBM ceiling — you may need TP=2 or H200; (5) memory leak — rare but seen with custom LoRA hot-swapping; check nvidia-smi --query-gpu=memory.used --format=csv -l 1.

Q: What's the typical KV cache hit rate for an agent workload?

A: Higher than chat. Agents tend to send repeated tool definitions, repeated system prompts, and repeated retrieval contexts across many steps in the same conversation. Production agent workloads on SGLang report 70-90% prefix cache hit rates after warm-up, vs 30-50% for typical chat traffic. The economic implication: agent serving cost per token of generated output is often 2-3x lower than chat cost per token because the prefill portion is largely cached. See agent serving infrastructure.

Q: How does the KV cache interact with reasoning model "thinking" tokens?

A: Thinking tokens accumulate KV like any other generated tokens. For models like DeepSeek-R1 that generate 5k-30k thinking tokens before the answer, the KV grows linearly through the thinking phase. Practical implication: serving R1-class models requires ~5-10x more per-request KV budget than non-reasoning peers, even for short user prompts. Most production stacks treat thinking tokens identically to answer tokens in the cache; some experimental work explores discarding thinking-token KV after the answer is produced, but this prevents follow-up reasoning. See reasoning-model serving.

Q: How does sliding-window attention change the KV math?

A: With window size W, each sequence's effective KV is bounded by W tokens, not by full context length. For Mistral 7B with W=4096: a 32k-context request only ever holds 4096 tokens of KV at any moment. The serving win is large at very long context — KV per request is constant beyond W rather than growing linearly. The quality cost: information beyond the window is dropped, which hurts long-context recall benchmarks. Gemma-2 alternates SWA layers and full-attention layers to balance this; Mistral's later releases moved to global attention with smarter compression. See long-context attention.

Q: What's the practical max-context for B200 serving in 2026?

A: With FP8 KV and a typical 70B-class GQA model, B200's 192 GB holds ~100 GB usable KV after weights and workspace, which is ~5M token-equivalents at 22 KB/token. The bottleneck shifts from capacity to compute (prefill at 5M tokens takes minutes) and bandwidth (attention compute over 5M positions is expensive). Production deployments cap at 1-2M context even when memory allows. Gemini 2.x at 2M context, Claude at 1M, and various Llama-3 long-context fine-tunes at 256k-1M represent the realistic frontier.

Q: Can the KV cache be precomputed for static system prompts?

A: Yes — this is exactly what prefix caching does. For an extreme case: precompute KV for a 50k-token system prompt once at server startup, mark those blocks as immutable, and every request implicitly starts with that KV loaded. SGLang and vLLM both support "precomputed prefix" workflows. The win is huge for use cases with long, static prefixes: agent tool definitions, RAG framework boilerplate, long few-shot example sets. The implementation gotcha: any change to the prompt invalidates the cache, so version your prefixes.

---

Glossary

• Attention sink: the first few tokens of a sequence, which softmax disproportionately attends to even when they're not semantically informative. Important for stream-friendly truncation strategies (StreamingLLM).
• Block table: a per-sequence mapping from logical positions to physical KV cache block IDs. The data structure that makes paging work.
• Continuous batching: prefilling new requests into an in-flight decode batch as they arrive, instead of waiting for a fixed batch size. The right default for production.
• EAGLE / EAGLE-2: speculative decoding approaches that share the target model's hidden states with a small draft head. Standard in 2026.
• Eviction: removing an in-progress sequence's KV from the cache to make room for new requests. Recompute and swap are the two main strategies.
• FlashAttention / FlashAttention-2 / FlashAttention-3: memory-efficient attention kernels. The default attention implementation in most stacks.
• FP4 / FP8 (e4m3 / e5m2): floating-point formats with 4 or 8 total bits. Used for KV quantization.
• GQA (Grouped-Query Attention): attention variant where multiple query heads share K and V heads. GQA-8 is the modern standard.
• HBM: high-bandwidth memory. The physical memory on a GPU.
• Internal fragmentation: per-request over-reservation. You allocated max_context_len; the request only used 2k tokens.
• External fragmentation: gaps between sequences that can't be reused for new sequences too large to fit in any single gap.
• LRU / LFU: cache eviction policies. Least Recently Used / Least Frequently Used.
• MEDUSA: speculative decoding via multiple parallel decoding heads on the target model. No separate draft model.
• MHA (Multi-Head Attention): original attention with one K and V head per query head.
• MLA (Multi-Head Latent Attention): DeepSeek's attention variant that caches a low-rank latent instead of full K and V.
• MoE (Mixture of Experts): architecture where each token is routed to a subset of expert MLP blocks. Active vs total parameters differ.
• MQA (Multi-Query Attention): attention with exactly one K and V head shared across all query heads.
• PagedAttention: KV cache management using fixed-size blocks plus a per-sequence block table. Standard since 2023.
• PCIe (Gen4 / Gen5): peripheral component interconnect. Bandwidth between CPU and GPU. Gen5 ~64 GB/s/direction.
• Prefill: computing KV for all the tokens of a prompt in parallel, before the first generated token.
• Prefix caching: deduping KV blocks across requests that share a prefix.
• RadixAttention: SGLang's prefix-sharing implementation using a radix tree.
• RoPE (Rotary Position Embedding): position encoding via rotation. Used in nearly all modern transformers.
• RULER: a long-context retrieval benchmark. Tests needle-in-haystack across multiple lengths.
• Speculative decoding: generating K candidate tokens with a draft model, verifying with the target model.
• SSM (State Space Model): alternative to attention with constant per-step memory. Mamba, Mamba-2 are leading examples.
• SWA (Sliding-Window Attention): attention with a fixed-size window. KV cache is bounded by window size.
• TP (Tensor Parallelism): splitting weight matrices across multiple GPUs.
• TTL: time-to-live. A cache eviction policy.
• vLLM / SGLang / TRT-LLM / TGI / LMDeploy / llama.cpp: the main serving stacks in 2026.

---

References

• Vaswani et al., Attention Is All You Need, NeurIPS 2017. The original Transformer.
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, 2019. (MQA)
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023.
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022.
• Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, 2023.
• Shah et al., FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision, 2024.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. (vLLM)
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, NeurIPS 2024. (RadixAttention)
• DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024. (MLA)
• DeepSeek-AI, DeepSeek-V3 Technical Report, 2024.
• Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.
• Hooper et al., KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, NeurIPS 2024.
• Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024. (StreamingLLM)
• Dao & Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, ICML 2024. (Mamba-2)
• Lieber et al., Jamba: A Hybrid Transformer-Mamba Language Model, AI21, 2024.
• Li et al., EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, 2024.
• Cai et al., MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024.
• Fu et al., Lookahead Decoding: A Decoding Algorithm for Faster LLM Inference, 2024.
• Beltagy et al., Longformer: The Long-Document Transformer, 2020. (sliding window + global attention)
• Zaheer et al., Big Bird: Transformers for Longer Sequences, NeurIPS 2020.
• Kitaev et al., Reformer: The Efficient Transformer, ICLR 2020. (LSH attention)
• DeepSeek-AI, Native Sparse Attention, 2025. (NSA)
• Peng et al., RWKV: Reinventing RNNs for the Transformer Era, 2023.

---