Modern LLM Decoding: Speculative, Lookahead, Medusa, EAGLE — The Complete Guide

Every advertised "tokens per second" number on a serving dashboard is a function of one architectural choice: how the model decides what the next token should be. For a long time the answer was boring — sample one token, append, repeat. Modern decoding is no longer boring. The decode loop has been rewritten three times in three years, and the layer that produces tokens is now where most of the inference engineering effort sits.

This guide is the authoritative answer to how modern LLM decoding actually works in 2026. We start from the basics — greedy, sampling, beam — and move through the structural problem (decode is memory-bandwidth-bound, not compute-bound) into the cluster of techniques that solve it: vanilla speculative decoding (Leviathan/Chen 2023), Medusa, EAGLE and EAGLE-2, Lookahead, self-speculative decoding via early exit (LayerSkip), retrieval-based REST, and the draft-model strategies that make them work in production. Speculative decoding is the headline because it has the cleanest theoretical guarantee — the output distribution is provably identical to vanilla decoding — but it is only the most-used member of a larger family. Production stacks now compose 2-3 of these techniques on top of continuous batching, paged attention, and disaggregated prefill/decode, which is where the real end-to-end speedups come from.

Speculative decoding flips the wasteful default: generate multiple candidate tokens cheaply (small draft model, draft head, or the target in lookahead) and verify them in one expensive target pass. Under speculative sampling (Leviathan / Chen 2023, rejection-sampling step), the output distribution is provably identical to vanilla decoding — no quality loss, pure speedup.

Opinionated defaults: EAGLE-2 for most large-target deployments, Medusa if you control training, Lookahead if you want zero new infrastructure.

Key takeaways
Mental model: speculative decoding in one minute
The decoding landscape in 2026
Why decode is slow
The core algorithm
Variants: vanilla, EAGLE, MEDUSA, Lookahead, REST
Draft length K: tuning
KV cache implications
When spec-decode wins
When it doesn't
Stack support and configuration
Worked examples
Recent research directions
The bottom line
FAQ
Glossary
References
EAGLE-3 in detail
MEDUSA-2 and self-distillation
Lookahead Decoding and Jacobi iteration
REST: retrieval-based speculative decoding
BiLD: Big-Little Decoder
SpecInfer and tree-structured verification
PASS: pipeline-parallel speculative decoding
Per-stack support deep dive
Chunked prefill interaction
Disaggregated prefill/decode interaction
MoE target with speculative decoding
Reasoning models and speculative decoding
Structured output + speculative decoding
Multimodal targets
CPU and edge speculative decoding
Acceptance rate by position-in-sequence
FP8 and quantised drafts
Failure modes
Speedup math
Production case studies (2026)
Production defaults table
Extra FAQ for 2026
Glossary additions for 2026
Cross-references
Benchmark deep dive: Llama-70B across workload classes
How EAGLE works inside
How Medusa works inside
How Lookahead works inside
Production checklist
Where speculative decoding doesn't help
The 2027 outlook

Key takeaways

Speculative decoding amortizes the cost of one large-model forward pass across multiple tokens by drafting K candidate tokens with a small model and verifying them in one target-model pass.

Vanilla (Leviathan/Chen 2023): independent draft model, simple, requires having a small model.
EAGLE-2 (2024): the dominant variant in 2026. Draft is a small head sharing target's hidden states. Free compute, ~10–15% extra KV.
MEDUSA (2024): multiple decoding heads on the target model. No separate draft. Simpler operationally.
Lookahead decoding (2024): the target model itself drafts via lookahead. No draft model.

Throughput wins (typical):

2–3× on agentic workloads (high token-prediction confidence).
1.5–2.5× on chat.
1.0–1.3× on creative writing (low predictability).

The math: spec-decode produces outputs statistically identical to vanilla decoding. Quality is preserved exactly. The speedup is unconditional.

Mental model: speculative decoding in one minute

The problem has a name: the autoregressive bottleneck. A transformer in decode mode does one forward pass per token. Each pass reads the full weight tensor from HBM — 140 GB for Llama-3 70B — to produce a single token. At batch 1, the GPU's tensor cores sit ~95% idle while HBM bandwidth is the bottleneck. The work per step is tiny; the memory traffic per step is enormous. You are paying for an H100 to do the data-movement equivalent of a register copy.

The fix is draft-and-verify. A cheap module (small draft model, extra heads, or a partial-depth forward) proposes K candidate tokens. The expensive target model then evaluates all K in one forward pass — at the same memory cost as a single decode step, because the weights are read once. A rejection-sampling step (Leviathan/Chen 2023) keeps only the prefix that the target would have produced anyway, so the output distribution is provably identical to vanilla sampling. The analogy is a typist with predictive autocomplete: the assistant suggests the next phrase, the typist accepts the prefix that's right, fixes the first wrong character, and moves on.

Without spec-decode vs with spec-decode:

Aspect	Vanilla decode	Speculative decode
Forward passes per accepted token	1 target	1 target / (1 + accepted)
HBM bandwidth utilization	60–90% (saturated)	Same, but amortized over K tokens
Tensor-core utilization	~5%	~30–60% (verification batches K tokens)
Extra memory	None	Draft weights + draft KV (~1–10%)
Output distribution	Target	Provably identical to target
When it pays off	Never the bottleneck — substrate	Decode-bound (chat, agents, code)

Minimal pseudocode:

while not done:
    draft = small_model.generate(ctx, K=5)        # cheap
    logits = target_model.forward(ctx + draft)    # one expensive pass
    accepted = rejection_sample(logits, draft)    # exact-equivalence
    ctx += accepted                                # 1..K+1 tokens at once

Production one-liner (vLLM): --speculative-model eagle --num-speculative-tokens 5.

Sticky number: EAGLE-2 delivers 2.5–3× decode speedup on Llama-3 70B with provably identical output — and stacks on top of continuous batching and paged attention.

The rest of this guide is which variant to pick, how big K should be, what breaks at low acceptance rates, and how the major serving stacks expose it.

The decoding landscape in 2026

"LLM decoding" is shorthand for an entire menu of techniques. Most discussions collapse them into "speculative decoding vs not", which hides the real choice space. Before going deep on any one variant, here is the full landscape, in roughly the order it evolved.

1. Greedy decoding. Pick the argmax of the next-token logits. Deterministic, fast per step, low diversity. Still the right default for many structured-output and code-completion tasks where you want reproducibility.

2. Sampling (with temperature, top-k, top-p). Sample from a softened distribution. The standard for chat. Temperature controls how flat the distribution becomes; top-p (nucleus) and top-k truncate the tail. Acceptance rate of speculative methods later in this list depends sharply on temperature — see §5.

3. Beam search. Maintain k partial hypotheses, expand all, keep top-k by joint probability. Useful for translation and constrained generation. Largely abandoned for open-ended generation because it produces bland, mode-collapsed text and adds latency.

4. Vanilla autoregressive decode. One forward pass per token. Reads the entire model's weights from HBM every step. This is the workload that everything below is trying to fix, because at batch 1 it leaves ~95% of an H100's tensor cores idle.

5. Continuous batching. Not a decoding algorithm per se, but the substrate everything else assumes. vLLM, SGLang, and TRT-LLM admit new requests into a running batch token-by-token instead of waiting for the slowest sequence in a batch to finish. This is how decode utilization gets pushed from ~5% of peak to ~50% on production workloads. Required reading: our LLM serving guide.

6. Vanilla speculative decoding (Leviathan, Chen — 2023). Independent draft model proposes K tokens; target verifies in one pass; rejection sampling guarantees identical output distribution. The foundational technique.

7. SpecInfer / tree-based speculation (Miao et al. 2023). Drafts a tree of candidates instead of a chain; verification explores branches in parallel. Higher effective acceptance per verification call.

8. MEDUSA (Cai et al. 2024). Add K decoding heads on top of the target. Each head predicts position +1, +2, ..., +K. No separate draft model; verification still uses a tree-attention mask. Simpler to deploy than vanilla; requires fine-tuning the heads.

9. Lookahead decoding (Fu et al. 2024). The target itself drafts, by jointly producing the next token and several lookahead n-grams in parallel. No draft model, no extra heads. Modest speedup (1.2-1.5×) but zero new infrastructure.

10. EAGLE / EAGLE-2 (Li et al. 2024). A small "draft head" consumes the target's hidden states (not its outputs) to propose candidate trees. Higher acceptance than vanilla because the draft sees what the target was actually thinking. EAGLE-2 adds dynamic tree shape based on draft confidence. The dominant production variant in 2026.

11. Self-speculative decoding via early exit (LayerSkip, Elhoushi et al. 2024). The same model drafts using only its first N layers and verifies with all layers. No extra parameters at all. Useful when you cannot afford a draft model or extra heads (memory-tight serving, on-device).

12. REST (retrieval-based speculative decoding). Retrieve candidate continuations from a corpus instead of generating them with a model. Best for code, structured output, and workloads with high repetition.

13. Constrained / grammar-guided decoding. Outlines, llguidance, XGrammar, jsonformer — apply a grammar mask to logits to force valid JSON / regex / CFG output. Composes with speculative decoding (you draft inside the grammar). Increasingly important as agent and tool-use traffic grows.

14. Cascade and adaptive routing. Route easy tokens to a small model, hard tokens to a large one (FastChat-style cascades, hybrid routers). Different from speculation because outputs are not formally identical — quality depends on the router.

Where each technique fits

Technique	New compute	Extra params/KV	Typical speedup	Quality guarantee	When to pick
Greedy / sampling	None	None	1× (baseline)	Exact	Always available
Continuous batching	None	None	5-10× throughput	Exact	Always — substrate
Vanilla spec-decode	Run small draft	Draft model + KV	1.5-2×	Provably exact	Have a tokenizer-matched small model
SpecInfer (tree)	Tree verification	Same as vanilla	1.7-2.3×	Provably exact	Multi-tenant, high QPS
MEDUSA	K extra heads	Heads (~5-10% params)	1.5-2×	Exact (with proper sampling)	Training your own target
Lookahead	Slightly larger forward	None	1.2-1.5×	Exact	Zero-infra speedup
EAGLE-2	Tiny draft head + tree	~1-2% params, small KV	2-3×	Provably exact	Default in 2026
LayerSkip / self-spec	Partial-depth forward	None	1.5-2×	Exact (with proper sampling)	Memory-tight or on-device
REST	Corpus retrieval	Index	1.5-2× (code)	Provably exact	Code, structured output, RAG-adjacent
Constrained decoding	Grammar mask	Grammar	Neutral or +20%	Exact within grammar	Tool calls, JSON, agents
Cascade routing	Small + large	Two models	Workload-dependent	Approximate	Cost-sensitive, mixed traffic

The key division is between provably-equivalent techniques (numbers 6-12, which produce samples drawn from the target distribution) and approximate techniques (cascades, aggressive distillation). For most production work you want the former; the speedup is real, the math is honest, and you do not have to re-evaluate quality after deploying. For background on the related distillation techniques that cascades rely on, see our synthetic data and distillation guide.

These techniques compose. A modern serving stack typically runs continuous batching + paged attention + speculative decoding + grammar masks simultaneously, on a disaggregated prefill/decode topology, with FP8 KV cache. Each layer is independently a 1.3-2× win; the multiplicative stack is what gets advertised "10× throughput" numbers in serving benchmarks.

Why decode is slow

A modern GPU like H100 has:

~700 TFLOPs/sec sustained BF16 compute.
~3 TB/s HBM bandwidth.

The "arithmetic intensity" — FLOPs per byte of data moved — needed to keep tensor cores busy is roughly 230 FLOPs/byte.

A typical decode step on Llama-3 70B:

Reads the entire model: 140 GB.
Reads the KV cache for the request: ~10 MB at 32k context.
Generates one new token: ~140 GFLOPs.

Arithmetic intensity: 140 GFLOPs / 140 GB = ~1 FLOP/byte. Two orders of magnitude below what tensor cores need. Decode is bandwidth-bound, not compute-bound.

Concretely on Llama-3 70B FP8 on H100:

Achieved compute: ~30 TFLOPs/sec (4% of peak).
Achieved bandwidth: ~2.7 TB/s (90% of peak).

The GPU is saturated on memory reads but barely scratched on math. There's massive headroom in compute. Speculative decoding spends that compute on verifying candidate tokens.

The core algorithm

Pseudocode:

state = initial_kv_cache
while not done:
    # Step 1: draft K candidate tokens, cheaply
    candidates = []
    draft_state = state.copy()
    for i in range(K):
        token, draft_state = draft_model(draft_state)
        candidates.append(token)

    # Step 2: target model verifies candidates in ONE forward pass
    target_logits = target_model.forward(state, candidates)

    # Step 3: speculative sampling — accept the longest valid prefix
    accepted = []
    for i, candidate in enumerate(candidates):
        target_prob = target_logits[i].prob_of(candidate)
        draft_prob = draft_logits[i].prob_of(candidate)
        if random() < target_prob / draft_prob:
            accepted.append(candidate)
        else:
            # Reject — sample replacement from corrected target distribution
            replacement = sample_corrected_dist(target_logits[i], draft_logits[i])
            accepted.append(replacement)
            break  # All subsequent candidates re-drafted

    output.extend(accepted)
    state = update_kv_cache(state, accepted)

Key insight: the speculative sampling step is mathematically constructed so that accepted tokens are distributed identically to vanilla draws from the target model. This is the rejection-sampling trick from Leviathan et al. (2023).

Net effect: vanilla decoding and speculative decoding produce statistically identical sequences. The only difference is wall-clock speed.

Acceptance rate

Fraction of drafted tokens accepted on average. Higher = bigger speedup. Depends on:

Draft model quality: a draft tracking target predictions has high acceptance.
Workload predictability: code is more predictable than poetry.
Sampling temperature: higher temperature spreads target distribution, more drafts plausible.
Vocabulary alignment: draft and target must use the same tokenizer.

Typical numbers for Llama-3 70B target with Llama-3 8B draft:

Code completion: 80–90% acceptance.
Chat: 60–75%.
Creative writing: 40–55%.

Variants: vanilla, EAGLE, MEDUSA, Lookahead, REST

Vanilla speculative decoding (Leviathan 2023, Chen 2023)

The original (arXiv:2211.17192, arXiv:2302.01318). Draft model is a separate, smaller LLM (e.g., Llama-3 8B drafting for Llama-3 70B). Pros: works with any pair of compatible models. Cons: requires running two models in lockstep; the draft has its own KV cache and weights to load. See also SpecInfer for tree-based verification.

EAGLE / EAGLE-2 (Li 2024)

The dominant variant in 2026 (arXiv:2401.15077, arXiv:2406.16858). Instead of a separate draft model, EAGLE uses a small "draft head" that consumes the target model's hidden states (not its outputs). Much smaller than a full model.

Draft compute: minimal (essentially a single transformer layer per draft token).
Draft KV: small additional state.
Acceptance rate: typically higher than vanilla because the draft sees the target's internal representation.

EAGLE-2 generates a tree of candidates instead of a linear chain, letting verification explore multiple branches simultaneously. Higher average acceptance.

MEDUSA (Cai 2024)

Adds multiple decoding heads directly to the target model (arXiv:2401.10774). Each head predicts the token at position +1, +2, +3, etc., independently.

No separate draft model.
All compute is in the target.
Throughput win: 1.5–2× typical.

Simpler operationally (no two-model coordination) but lower peak speedup than EAGLE-2.

Lookahead decoding (Fu 2024)

Uses the target model itself for drafting via "lookahead" steps producing candidate n-grams (arXiv:2402.02057). No draft model, no extra heads. Just clever scheduling. A related variant, self-speculative decoding via early-exit (LayerSkip, arXiv:2404.16710), drafts using shallower layers of the same model.

Throughput win: 1.2–1.5×. Modest, but trivial to deploy.

REST (Retrieval-based)

A 2025 variant: instead of a model drafting, use retrieval over a corpus to suggest candidate continuations. Useful for code where the next likely tokens are often "similar to something in the codebase."

REST shines for code and structured output. For free-form text, worse than EAGLE-2.

Comparison

Variant	Draft compute	Extra KV	Speedup	Operational complexity
Vanilla	Full small model	Yes	1.5–2×	Two models
EAGLE-2	Small head	Small	2–3×	Tree search
MEDUSA	None	None	1.5–2×	Single model
Lookahead	None	None	1.2–1.5×	Trivial
REST	Retrieval lookup	None	1.5–2× (code)	Corpus index

Pick by workload

Default for production: EAGLE-2.
Simpler ops, smaller speedup OK: MEDUSA.
Don't want to set up draft: Lookahead.
Code-heavy workloads: REST.

EAGLE-3 and the 2026 frontier

EAGLE-3 (Li et al., late-2025 preprint) generalizes EAGLE-2 by training the draft head against a mixture of intermediate target hidden states, not just the last layer. The published acceptance gain over EAGLE-2 is 8-14% on chat, 4-9% on code, with the same KV footprint. SGLang shipped EAGLE-3 support in late 2025; vLLM 0.8+ added it behind a flag. On Llama-3.3-70B chat, the SGLang team reports EAGLE-3 produces 3.0-3.4x decode speedup over a non-spec baseline at batch 1, compared to 2.4-2.7x for EAGLE-2 on the same hardware (H100 SXM, FP8 weights, FP8 KV).

The other 2026 frontier is multi-token prediction (MTP) heads pretrained jointly with the target, popularized by DeepSeek-V3 and adopted by several frontier labs. MTP is closer to MEDUSA than EAGLE — extra prediction heads, no separate draft model — but the heads are baked in during pretraining, which gets meaningfully higher acceptance than fine-tuned MEDUSA heads. The serving side is identical to MEDUSA from a code path perspective; the win is purely in training.

Side-by-side acceptance benchmarks (Llama-3.3-70B, May 2026)

Measured on the SGLang public benchmark suite, H100 SXM, FP8 weights, T=0.7, K=5. Numbers are median acceptance rate across 1000 requests per workload.

Variant	Code (HumanEval)	Chat (LMSYS arena)	JSON tool calls	Creative (writingbench)	Decode speedup vs no-spec
Vanilla (1B draft)	62%	48%	71%	31%	1.6x / 1.4x / 1.9x / 1.1x
MEDUSA-2	71%	58%	79%	38%	1.9x / 1.6x / 2.1x / 1.2x
EAGLE-2	84%	71%	89%	52%	2.7x / 2.2x / 3.0x / 1.4x
EAGLE-3	89%	78%	92%	58%	3.1x / 2.5x / 3.4x / 1.6x
Lookahead	n/a	41% effective	n/a	30%	1.3x / 1.2x / 1.4x / 1.05x
REST (code-only)	91%	n/a	n/a	n/a	2.9x / n/a / n/a / n/a

Read: EAGLE-3 is the clear default in 2026 for new deployments; EAGLE-2 is still the right answer if your inference engine does not yet support EAGLE-3 stably. Lookahead remains the right answer when you literally cannot ship a new model artifact.

Draft length K: tuning

K = number of candidate tokens drafted per step.

The tradeoff

K too short (e.g., 2): not enough amortization. Verification overhead dominates.

K too long (e.g., 10+): draft accuracy drops with depth. Most candidates beyond position 5–6 are wrong.

Typical sweet spot: K=4–6 for most workloads.

How acceptance probability decays with K

Each subsequent draft token is conditional on previous draft tokens being correct. If acceptance rate is 70% per token, probability of accepting all K is 0.7^K:

K	Pr(all accepted)
2	49%
4	24%
6	12%
8	6%

But you don't need all-or-nothing — you accept the longest prefix. Average accepted tokens per step at K=6 with 70% per-token acceptance is ~3–4.

Auto-tuning

vLLM 0.7+ auto-tunes K based on observed acceptance rates. Recommended on by default for production.

Per-request K vs global K

The crude default ("K=5 for everyone") leaves throughput on the floor when traffic is heterogeneous. Most production stacks now support per-request K, set in three ways: (1) explicit client hint (x-spec-k: 7 header in vLLM's OpenAI-compatible server), (2) workload classifier infers K from the request shape (presence of tools=[...] argument bumps K to 7, plain chat stays at 4), (3) online adaptive: track per-session acceptance for the last N tokens, raise K when rolling acceptance > 80%, lower when < 55%. SGLang's --speculative-num-steps-auto does (3) natively.

K vs tree width tradeoff (EAGLE-2/3)

EAGLE-2 and EAGLE-3 do not have a single K — they have a tree with depth (analogous to K) and width (branches per level). A 4-deep tree with 4 branches per level evaluates 256 candidate paths in one verification pass. The defaults in SGLang are depth 5, width 8 at root, 2 at deeper levels, which is ~32 candidate paths and ~64 candidate tokens evaluated per verification call. Going beyond depth 7 rarely pays — draft confidence collapses, verification compute grows linearly, but accepted-token gains plateau because each extra depth contributes < 0.1 expected accepted tokens.

Acceptance rate by sequence position

Acceptance is not flat across a generation. Empirically (Llama-3.3-70B + EAGLE-2 draft, chat):

Tokens 1-32: 78% acceptance (model is following obvious continuation of prompt).
Tokens 33-128: 71%.
Tokens 128-512: 67%.
Tokens 512+: 64% (drift, model considering more open-ended directions).

Implication: TTFT-adjacent decode is cheaper than steady-state decode. If your workload is heavy on short responses (chat, agent steps), your effective speedup beats the long-context published numbers.

KV cache implications

Speculative decoding doesn't come for free in memory:

Target model KV

The target's KV grows by K tokens per verification step. At 32k context, peak in-flight KV during verification is (32k + K) × kv_per_token.

Capacity planning: size the KV pool for peak K, not average. Otherwise you'll OOM mid-step.

Draft model KV (vanilla and EAGLE)

Vanilla: 10–15% extra for the draft. EAGLE: 2–5% extra.

Combined with prefix caching

Prefix caching and spec-decode compose. The prefix cache works on the verified KV state; spec-decode operates downstream of cache lookups. No special interaction required.

Combined with FP8 KV

Same KV format applies to both target and draft. FP8 KV with spec-decode is the standard production combination in 2026.

When spec-decode wins

The question: how predictable is the next token given prior context?

Highly predictable workloads (2–3× speedup)

Code completion: lots of mechanical patterns. Acceptance 80–90%.
Tool/function calls: structured output with predictable schema. Acceptance often >85%.
Repeated context: agents that re-emit similar reasoning across iterations.
JSON / structured output: highly constrained, near-deterministic.

Moderately predictable (1.5–2× speedup)

Chat with consistent style: most user-assistant chat. Acceptance 60–75%.
Summarization: somewhat predictable framing.
Translation: predictable structure within sentences.

Low-predictability (<1.5×, sometimes <1×)

Creative writing with high temperature: each token is a coin flip.
Reasoning at high entropy points: model considering multiple plausible answers.
Very small models (under 7B): target and draft too close in capability.

In low-predictability workloads, draft generation overhead can exceed savings. Net throughput drops.

Rule of thumb

Below ~50% acceptance rate, spec-decode is roughly break-even. Above ~70%, it's a clear win. Measure your acceptance rate, then decide.

When it doesn't

Beyond low predictability, spec-decode can fail when:

Memory-constrained deployments

The draft's extra KV may force you to reduce concurrent requests. If you're already KV-bound, throughput from "more concurrent" might exceed per-request speedup. Profile both.

Very small target models (under 30B)

Draft and target are too close in capability. Draft's per-token cost approaches target's, eating amortization. Spec-decode is for big targets.

Production multi-tenant with diverse workloads

If your workload mixes high-acceptance and low-acceptance traffic, the auto-tuner has trouble. Some tenants get big wins; others see slowdowns.

Speculative loops in agents

Reasoning models like o1 / R1 sometimes don't benefit much from spec-decode because the thinking is exploratory.

Stack support and configuration

vLLM

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --use-v2-block-manager

For EAGLE specifically:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --speculative-config '{"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3-Instruct-70B"}'

SGLang

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.3-70B-Instruct \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path yuhuili/EAGLE-LLaMA3-Instruct-70B \
  --speculative-num-steps 5

TensorRT-LLM

Spec-decode is built into the engine. Configure during build:

trtllm-build --speculative_decoding_mode lookahead \
  --max_input_len 8192

TGI

Supports vanilla speculative decoding via --speculate K flag.

LMDeploy

Supports vanilla and EAGLE via --speculative-decoding.

Stack maturity matrix

Stack	Vanilla	EAGLE	MEDUSA	Lookahead
vLLM	✅	✅	⚠ partial	✅
SGLang	✅	✅	✅	✅
TRT-LLM	✅	✅	✅	✅
TGI	✅	⚠ early	❌	❌
LMDeploy	✅	✅	❌	❌

Worked examples

Example 1: code-completion service

Llama-3 70B target with Llama-3 8B draft. Code completion, average 200 output tokens.

Without spec-decode: ~30 tokens/sec/request.
Acceptance rate measured: 87%.
K=6.
Average accepted per step: ~5.

Effective speedup: ~3×. From 30 to 90 tokens/sec/request.

Example 2: customer support chat

Llama-3 70B + Llama-3 8B draft. Chat, 150 output tokens.

Acceptance rate: 65%.
K=4.
Average accepted per step: ~2.5.
Effective speedup: ~1.8×.

Example 3: creative writing

Llama-3 70B. Story-writing with high temperature (0.9).

Acceptance rate: 45%.
K=3.
Average accepted per step: ~1.4.
Effective speedup: ~1.1×.

For high-entropy creative workloads, skip spec-decode.

Example 4: agent / tool use

Llama-3 70B as agent. Heavy structured output (JSON tool calls).

Acceptance rate: 80%.
K=6.
Average accepted per step: ~4.5.
Effective speedup: ~2.5×.

Agents are the killer use case for spec-decode.

Recent research directions

Tree-based drafts

EAGLE-2 already does this. Future variants explore deeper tree expansion (8+ levels) with smarter pruning.

Cross-request speculation

Use successful drafts from one request as priors for similar requests. Early research; not in production stacks yet.

Adaptive K within a single request

K varies token-by-token based on local entropy estimates. Some 2025 papers show 10–15% additional speedup.

Speculative decoding for reasoning models

Specialized variants in development for o1/R1-style reasoning models.

Quantized drafts

The draft model itself can be quantized aggressively (INT4) for further speedup. Acceptance rates drop slightly, but savings on draft cost can dominate.

A short history of speculative decoding

Speculative decoding emerged from the simultaneous discovery that the speedup mathematics were possible. A timeline:

2018: original "speculative parallelism" ideas in compiler optimization. Not yet applied to LLMs.

2022: Stern et al. publish "Blockwise Parallel Decoding for Deep Autoregressive Models" — the first concrete proposal for parallel decoding via auxiliary heads.

2023 (early): Leviathan et al. publish "Fast Inference from Transformers via Speculative Decoding" — the seminal paper. Independent, simultaneous publication: Chen et al. "Accelerating Large Language Model Decoding with Speculative Sampling."

2023 (late): implementations in major inference engines. vLLM, TGI, TRT-LLM all integrate basic speculative decoding.

2024 (early): MEDUSA (Cai et al.) introduces multi-head speculative decoding as a simpler alternative.

2024 (mid): EAGLE (Li et al.) introduces feature-level speculation. Higher acceptance rates than vanilla.

2024 (late): EAGLE-2 with dynamic tree drafting. Becomes the dominant variant.

2025: tree-based speculation, lookahead decoding, REST, and various hybrids.

2026: speculative decoding is standard in production stacks. Auto-tuning is the default.

The pattern: each variant chips away at the same core insight. Different trade-offs but the same fundamental speedup.

Mathematical correctness

Why speculative decoding produces statistically identical output to vanilla decoding.

Rejection sampling derivation

The acceptance criterion at each position:

accept candidate c with probability min(1, p(c) / q(c))

where p(c) is target's probability for c, q(c) is draft's probability for c.

If accepted: output is c. If rejected: sample replacement from corrected distribution: (p - q) / (1 - q_total_rejected).

This guarantees: the marginal distribution of accepted tokens matches target's distribution exactly.

Why both probabilities are needed

The draft probability q(c) is not just for the rejection ratio — it's also for the "corrected distribution" formula. Without it, you couldn't sample replacement tokens correctly.

This is why speculative decoding requires the draft model to output proper probabilities, not just discrete tokens.

Temperature interaction

At temperature T, both target and draft probabilities are softened: p_T(c) = p(c)^(1/T) / Z.

Higher temperature → flatter distributions → higher acceptance rate (more tokens are "plausible" under both models).

This is why high-temperature workloads (creative writing) sometimes have lower acceptance: the draft and target distributions diverge more when both are flat.

Quality preservation guarantee

The math:

Target distribution: P(c|context).
Draft distribution: Q(c|context).
After speculative decoding: distribution of accepted tokens = P(c|context).

This is a strong guarantee — not approximate, not statistical, but exact under the rejection sampling derivation.

The only quality loss in practice comes from:

Implementation bugs.
Reduced batch size for memory reasons.
Numerical errors in probability computation.

A correctly-implemented speculative decode produces samples indistinguishable from vanilla.

Architecture deep dive: EAGLE-2

EAGLE-2 is the most-deployed variant in 2026. How it works.

Draft model architecture

The draft is a single transformer "head" that takes the target's hidden states and predicts the next token.

target_hidden_state[i] → draft_layer → next_token_prediction

The draft layer is small: roughly 1-2% of the target model's parameters. For Llama-3 70B target, the draft is ~700M parameters.

Tree-structured drafting

Instead of a linear chain (token1 → token2 → ... → tokenK), EAGLE-2 generates a tree:

                token1
              ↙   ↓   ↘
         token2a token2b token2c
            ↙↘     ↙↘     ↙
           ...

Each branch is a possible continuation. The verification pass evaluates all branches in parallel. Accepts the branch matching the target's distribution.

Result: more candidates explored per verification step. Higher effective acceptance rate.

Dynamic tree shape

EAGLE-2's "dynamic" part: the tree shape adapts based on draft confidence. Confident branches get expanded deeper; uncertain branches get cut short.

This vs static-shape tree drafting (Specinfer, Sequoia): dynamic adapts to the actual workload, static is simpler but less effective.

Verification batching

The target model processes the entire tree in one batched forward pass. Tokens at different positions are processed in parallel using attention masks that respect the tree structure.

This is the key efficiency: one expensive target forward pass amortizes across many candidate tokens.

Acceptance rate empirics

EAGLE-2 typically achieves 70-90% acceptance on chat workloads, 80-95% on code, 60-75% on creative writing.

Compared to:

Vanilla speculative decoding: ~50-70% acceptance.
MEDUSA: 60-80% (similar to EAGLE on some workloads).

EAGLE-2's edge comes from feeding the target's hidden states into the draft, not just past tokens.

Architecture deep dive: MEDUSA

MEDUSA's approach is simpler than EAGLE: instead of a separate draft model, add multiple "decoding heads" to the target.

Multiple heads

The target model gets K extra decoding heads. Each head predicts the token at position +1, +2, ..., +K.

hidden_state → standard_head → token at position +1
            → medusa_head_1 → token at position +2
            → medusa_head_2 → token at position +3
            → medusa_head_3 → token at position +4

All K predictions happen in one forward pass.

Verification

The K predictions are speculative; need verification. MEDUSA uses tree attention to verify all K candidates simultaneously, similar to EAGLE-2.

Trade-offs vs EAGLE

MEDUSA pros:

No separate draft model.
Fewer total parameters.
Easier to deploy.

MEDUSA cons:

Lower peak acceptance rate (60-80% typical).
Can't be added post-hoc — heads are trained jointly with the target.
Quality on existing target models requires fine-tuning.

For organizations training a new target model, MEDUSA is a natural choice. For deploying a pre-existing target, EAGLE-2 is more flexible.

Lookahead decoding

Lookahead is the simplest speculative variant: no separate draft, no extra heads. Just clever scheduling.

Mechanism

At each step, the target model generates K parallel "lookahead" predictions in addition to the normal next-token prediction.

Iteration N:    generate token N
                generate lookahead candidates N+1, N+2, ..., N+K (in parallel)
                
Iteration N+1:  use lookahead N+1 if it matches what target would predict
                otherwise: regenerate

The lookahead candidates are produced by the target itself, in a single forward pass that processes K positions simultaneously.

Speedup

Modest: 1.2-1.5× typically. Better than nothing, requires no architectural changes.

When to use

For deployments where:

Speculative decoding's peak speedup isn't worth integration complexity.
You don't have a draft model and don't want to maintain MEDUSA training.
Latency improvement matters but throughput cost is constrained.

Lookahead is the "easy mode" of speculative decoding.

REST: Retrieval-based speculative decoding

REST replaces the draft model with retrieval over a corpus.

Mechanism

At each step:

Hash the recent context.
Look up matching n-grams in a pre-built index.
Use the most-frequent continuation as the candidate.
Verify with target model.

When REST shines

For workloads where the next likely tokens are "similar to something seen before":

Code: repeated patterns across codebases.
Boilerplate: lots of mechanical structure.
Templated output: structured generation that follows known patterns.

For these, REST's acceptance rate exceeds EAGLE-2's. The retrieval is more precise than what a small draft model can predict.

When REST fails

For workloads with novel or creative output, retrieval finds nothing useful. REST degrades to no speedup.

Index size and quality

REST's quality depends on the retrieval index:

Larger corpus → more matches → higher acceptance.
More relevant corpus → higher quality → higher acceptance.
Updating the corpus needs recomputation of the index.

For code, indexing the codebase you're working on is a natural fit. Some IDE-integrated LLMs use REST-style retrieval.

Production deployment patterns

How real production deployments use speculative decoding in 2026.

Pattern 1: chat at scale

vLLM with EAGLE-2:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model yuhuili/EAGLE-LLaMA3-Instruct-70B \
  --num-speculative-tokens 5

Typical: 1.8-2.5× throughput improvement on chat workloads.

Pattern 2: code completion

vLLM with EAGLE-2 (code-specialized draft if available):

vllm serve codellama/CodeLlama-70b-Instruct-hf \
  --speculative-model some-code-eagle-draft \
  --num-speculative-tokens 6

Code is more predictable; longer K and higher acceptance. 2-3× speedup typical.

Pattern 3: agent workloads

Heavy tool calls and structured output. Highest acceptance rates of any common workload.

EAGLE-2 with K=5-7. 2.5-3× speedup typical. The structured nature of agent output makes drafts very accurate.

Pattern 4: reasoning models

For o1/R1-style models with long thinking sequences.

Spec-decode helps less here because thinking has higher entropy. But still 1.3-1.5× improvement.

Pattern 5: multi-tenant with heterogeneous workloads

If you have a mix of high-acceptance (code, agents) and low-acceptance (creative writing) traffic, the auto-tuner adapts K per request.

vLLM 0.7+ supports this dynamically. SGLang has similar.

The bottom line

The autoregressive bottleneck is the reason a $40k GPU spends most of its life moving weights from HBM to do almost no math. Decode is memory-bandwidth-bound, not compute-bound, and no amount of FLOPs will fix it. Speculative decoding is the only family of techniques that addresses this directly: amortize one expensive weight read across multiple accepted tokens by drafting cheaply and verifying in parallel. The single biggest lever is acceptance rate — every percentage point compounds against draft cost, and acceptance is what separates a 1.2× win from a 3× win.

If you take only this away:

Decode is bandwidth-bound at batch 1. Adding compute does nothing. The fix is fewer forward passes per token, not faster passes.
EAGLE-2 is the 2026 default for large open-weight targets — 2.5–3× with ~1–2% extra params and provably identical output.
MEDUSA wins when you control training; its no-draft-model operational simplicity beats EAGLE-2 in many shops.
Lookahead is the zero-infra speedup (1.2–1.5×) when you cannot ship new weights.
Speedup is workload-shaped. High-predictability traffic (code, agents, repeated boilerplate) hits 3×; creative writing barely moves.

For the substrate this sits on, read LLM serving and disaggregated prefill/decode — most of the multiplicative throughput you see in benchmarks comes from composing all three.

FAQ

Q: Does speculative decoding hurt model quality?

No. The math guarantees output is statistically identical to vanilla decoding.

Q: What's the difference between EAGLE and EAGLE-2?

EAGLE is the original (linear chain). EAGLE-2 adds dynamic tree search. Higher peak speedup.

Q: Should I use EAGLE or MEDUSA?

EAGLE for higher speedup; MEDUSA for simpler ops. Both are exact (no quality loss).

Q: How do I pick K?

Start with K=4. If high acceptance, bump to 5–6. If low, drop to 2–3. Better: enable auto-tuning.

Q: Does spec-decode help with prefill?

No. Spec-decode is decode-phase. Prefill is already compute-bound.

Q: Can I use a different model architecture as draft?

Theoretically yes (tokenizer must be identical). Practically: draft and target almost always come from the same family.

Q: How does spec-decode interact with batching?

Each request runs its own spec-decode. The batched verification pass amortizes target cost across requests. They compose.

Q: Memory cost?

Vanilla: 10–15% extra. EAGLE-2: 2–5%. MEDUSA: ~0%. Plan KV capacity for the worst case.

Q: Why doesn't every API just enable spec-decode?

Most do, by 2026. Frontier providers don't disclose, but throughput patterns suggest yes. Open-source stacks have it on by default in v0.7+.

Q: Does spec-decode work with reasoning models?

Yes, but acceptance rates can be lower in the "thinking" phase due to high entropy.

Q: Is spec-decode the same as parallel decoding?

No. Parallel decoding generates tokens for multiple positions in parallel via causal masks. Spec-decode generates and verifies. They're orthogonal.

Q: How do I train an EAGLE draft head?

Standard: train on the target model's hidden states. Loss: predict the target's next token given target's hidden state.

# Pseudocode
for batch in training_data:
    with torch.no_grad():
        target_hidden = target_model.get_hidden_states(batch)
    
    pred = eagle_head(target_hidden)
    loss = cross_entropy(pred, batch.next_tokens)
    loss.backward()

Training takes ~hours on a small dataset. Result: EAGLE-2 head specific to your target model.

Q: Can I use spec-decode with a quantized target?

Yes. The target's quantization (FP8 weights, INT4 weights) doesn't affect spec-decode mechanics. Target produces probabilities in FP32 internally; spec-decode uses those.

Performance gain: same percentage as un-quantized.

Q: Should I train EAGLE-2 specifically for my workload?

Possibly. The default EAGLE-2 heads are trained on diverse data. For specialized workloads (code, agents), workload-specific training improves acceptance by 5-15%.

Cost: hours of training. Worth it if your workload sees billions of inferences.

Q: How does spec-decode interact with multi-LoRA?

Mixed: works in principle but adds complexity. Each LoRA potentially needs its own draft head if quality matters.

Common simplification: use the base model's draft for all LoRAs. Acceptance rate slightly worse but operationally simpler.

Q: Does spec-decode help with multimodal models?

Yes. Image-to-text generation is highly predictable (caption-style output). Spec-decode acceptance rates similar to chat.

For multimodal input → text output, no special handling needed.

Q: What's the difference between speculative decoding and beam search?

Beam search: explores multiple candidate sequences in parallel, keeps top-K. Used for non-greedy decoding. Doesn't speed up; explores quality space.

Spec-decode: generates one sequence faster via amortization. Speed-up technique.

They can compose but rarely do in production (beam search is rarely used for chat anyway).

Q: Do hosted APIs use speculative decoding?

Most likely yes. Frontier providers don't disclose, but throughput patterns suggest spec-decode-style optimizations are universal.

OpenAI, Anthropic, and Google's APIs likely use proprietary spec-decode variants.

Q: How is spec-decode evaluated?

Standard: throughput on a representative workload. Metrics:

Tokens/sec/request (decode-phase only).
Acceptance rate average.
End-to-end latency.

Compare to non-spec-decode baseline. Anything below 1.5× speedup probably isn't worth the complexity.

Q: Can I use multiple draft models?

Theoretically yes (ensemble drafting). In practice, rare. The added complexity rarely justifies the additional speedup.

Q: How does spec-decode compose with other optimizations?

Compatible with: paged attention, prefix caching, FP8 KV, multi-LoRA, continuous batching.

The optimizations layer. EAGLE-2 + paged + prefix caching + FP8 KV + continuous batching = standard production stack in 2026.

Q: Is spec-decode useful for embedding models?

No. Embedding models have a single forward pass; no decoding loop. Spec-decode is decode-specific.

Q: What about beam search with spec-decode?

Some research extends spec-decode to beam search. Verifies multiple beams in parallel. Useful for translation and other non-greedy applications.

Production deployment is rare; beam search itself is uncommon for chat.

Q: Memory cost vs throughput gain — when is spec-decode worth it?

Calculation:

KV memory cost: 10-15% extra (vanilla) or 2-5% (EAGLE-2).
Throughput gain: 1.5-3× typical.

If your KV is the binding constraint and you'd lose >15% concurrency to spec-decode's memory overhead: not worth it.

If your KV has slack: definitely worth it.

Q: Will spec-decode become unnecessary as hardware improves?

No. Decode is fundamentally memory-bound. Faster hardware doesn't help; memory bandwidth limits.

Spec-decode amortizes that bandwidth across multiple tokens. The technique remains valuable as long as decode is bandwidth-bound, which is forever.

Q: How do I handle errors in spec-decode?

Catch exceptions in the verification step. Fall back to standard decode for failed batches. Most stacks handle this automatically.

Q: What's the best K for chat workloads in 2026?

K=4 is the safe default. K=5-6 if your chat traffic is highly templated. Auto-tuning is the right answer when available.

Q: When should I use speculative decoding vs MEDUSA vs Lookahead?

Decision tree:

Have a tokenizer-matched small model already: vanilla speculative decoding is the cheapest start. Ship in a day.
Training your own target model from scratch: bake MEDUSA heads in. They are free at inference and require no separate draft.
Cannot afford any extra infrastructure: Lookahead decoding. It is a flag in vLLM, costs nothing, and buys you 20-50%.
You operate a hosted, high-QPS large-target service: EAGLE-2. The tree drafting + hidden-state-aware draft head delivers the highest acceptance rate, and the draft head is small enough that the extra KV is rounding error.
Memory-bound (long context, multi-tenant): self-speculative decoding via early exit (LayerSkip). Zero extra parameters, fits inside your existing serving GPU.
Code or structured output, with a large code corpus available: REST as the drafter. Acceptance rates on copy-modify workloads exceed any neural draft.

The decisions are not exclusive — vLLM and SGLang both let you swap drafters per request.

Q: Does speculative decoding actually compose with disaggregated prefill/decode?

Yes, but with one wrinkle: the draft model has to live somewhere. The two clean options are (1) put the draft on the decode worker — adds memory pressure but keeps latency low, and (2) put the draft on a separate pool, which adds a network hop per draft step. Production stacks almost always pick (1). For more on the underlying serving topology, see our disaggregated inference guide.

Q: How does speculative decoding interact with MoE serving?

Surprisingly cleanly. MoE decode is even more bandwidth-bound per active expert than dense decode (you read different experts for different tokens), so the headroom for speculation is bigger. Acceptance rates are slightly lower because the expert-routing pattern adds entropy. EAGLE-2 with a dense draft head against an MoE target is the common production pattern. See our mixture-of-experts serving guide.

Q: What is the realistic ceiling on decoding throughput improvement?

The theoretical maximum from speculative decoding is bounded by 1 + E[accepted tokens per verification step], which is hard to push past ~4× even with perfect drafts because each verification call still has fixed cost. Stacking speculative decoding with continuous batching, FP8 KV cache, and disaggregation gets you another 2-3× on top, for a realistic 5-8× over a naïve PyTorch baseline. Anything beyond that requires either model-level changes (Mamba, hybrid attention from the long-context guide) or hardware-level changes (B200 / GB200).

Q: Does the draft model need to be the same family as the target?

It needs to share a tokenizer. Same family is the easy way to guarantee that. Cross-family speculation (e.g., a Qwen draft for a Llama target) works only if you align tokenizers, which is brittle and rarely worth the engineering. The standard pattern is Llama-3.x 1B/8B drafts for Llama-3.x 70B/405B targets.

Q: How do EAGLE-2 and EAGLE-3 differ in practice?

EAGLE-3 conditions the draft head on a mixture of intermediate hidden states from multiple target layers, rather than just the last layer's hidden state. The acceptance gain is 5-10 percentage points on chat and 3-6 percentage points on code, with no change in KV footprint. The cost is training: the EAGLE-3 head needs to see hidden states from at least 3 sampled target layers during training, which roughly doubles draft-training time. For inference there is no operational difference — same API, same flags, swap the artifact.

Q: What is the right draft model size for a 70B target?

Empirically, a 1B EAGLE-2 head outperforms a 7B vanilla draft for the same target. For Llama-3.3-70B the sweet spot is the published yuhuili/EAGLE3-LLaMA3.3-Instruct-70B head at ~1.2B parameters. For vanilla speculation without a trained draft head, Llama-3.2-1B is the right size — Llama-3.1-8B is overkill for drafting and burns too much KV. The general rule: draft compute should be 1-3% of target compute per token; below that, the draft is too weak; above that, you can't amortize.

Q: Does FP8 quantization of the draft hurt acceptance?

Slightly. FP8 weights on the draft drop acceptance by 1-2 percentage points relative to BF16 draft for the same target. INT4 weights drop acceptance by 3-5 points. The compute savings on the draft side usually outweigh the acceptance loss because draft FLOPs become near-free, allowing slightly larger K. The standard 2026 production pattern is FP8 target, FP8 draft, FP8 KV — see the quantization tradeoffs guide.

Q: How does spec-decode interact with chunked prefill?

They are orthogonal. Chunked prefill splits the prefill compute into smaller batches to overlap with decode in continuous batching; spec-decode operates entirely in the decode phase. The interaction worth knowing: when a long prompt is being chunk-prefilled, decode tokens for other in-flight requests are running spec-decode against a target that is simultaneously doing prefill. Verification calls compete with prefill chunks for tensor-core time. vLLM's scheduler prioritizes verification batches over prefill chunks when decode tokens are nearly complete; SGLang inverts this. Profile.

Q: When is REST better than EAGLE-3?

When the next likely tokens literally exist verbatim in a known corpus. The clearest wins: (1) coding agents working inside a specific repo (index that repo), (2) document Q&A where the answer paraphrases retrieved chunks, (3) templated structured output like SQL or specific JSON schemas. For these, REST acceptance routinely hits 90-95%, beating EAGLE-3's 85-92%. For everything else, EAGLE-3 wins because retrieval finds nothing useful in open-ended generation.

Q: Can I run spec-decode without retraining anything?

Yes, via vanilla speculation with an off-the-shelf small model from the same family (e.g., Llama-3.2-1B drafting for Llama-3.3-70B). Or Lookahead, which uses the target itself. Or LayerSkip self-speculation if the target was trained with early-exit capability. EAGLE / EAGLE-2 / EAGLE-3 / MEDUSA all require a trained artifact, though publicly trained heads exist for the major Llama, Qwen, and Mistral checkpoints.

Q: How does spec-decode affect tail latency (p99)?

Mean latency drops significantly (1.5-3x). p99 latency drops less reliably. The variance source: occasional batches where acceptance collapses (high-entropy region of generation) and the verification step did all the target-model work for almost no accepted tokens. SGLang and vLLM both let you cap K dynamically, which clamps the worst case. The realistic expectation: p99 drops 30-60% with spec-decode, vs. 50-67% drop in mean.

Q: How does spec-decode interact with disaggregated prefill/decode?

The draft model lives on the decode worker. Prefill workers run standard, non-speculative attention. KV transfer from prefill to decode is unaffected — only the verified KV is transferred, and verification happens on the decode side. The one wrinkle is that EAGLE/EAGLE-3 drafts consume target hidden states from intermediate layers; if your decode worker uses tensor-parallel sharding (TP > 1), the draft head needs to gather hidden states across shards before drafting, adding 5-10 microseconds of NCCL traffic per draft step. See the disaggregated inference guide for the broader topology.

Q: What is the operational risk of EAGLE-3 vs EAGLE-2?

EAGLE-3 was released in late 2025; the public heads have had ~6 months of production exposure as of mid-2026. The known failure mode: numerical instability in the multi-layer hidden-state mixing under aggressive FP8 quantization of the target. Mitigation: run draft head in BF16 even when target is FP8. vLLM and SGLang both default to this. The known good config: Llama-3.3-70B target FP8, EAGLE-3 draft BF16, FP8 KV. Production-stable since November 2025.

Q: Does spec-decode help with MoE targets like Mixtral or DeepSeek-V3?

Yes, often more than dense targets. MoE decode is bandwidth-bound on expert weights, with the additional cost of routing entropy. A speculative verification pass on an MoE target activates a small superset of experts per position, but reuses the same experts when consecutive draft tokens route similarly. Acceptance is typically 3-5 points lower than for a dense target of equivalent capability because router outputs add randomness. Net throughput win: 2.0-2.6x, vs. 2.5-3.0x for dense. See the mixture-of-experts serving guide.

Q: Can spec-decode break grammar-constrained decoding?

It can, if the draft proposes tokens that violate the grammar. The clean fix is to apply the grammar mask both to the draft (so it never proposes invalid tokens) and to the target during verification. SGLang's xgrammar integration handles this end-to-end. vLLM's guided_decoding does the same with outlines or lm-format-enforcer. Acceptance rates on grammar-constrained workloads are usually higher than unconstrained, not lower, because the grammar narrows both distributions to the same valid subset.

Q: What's the failure mode when acceptance silently collapses?

Three common causes: (1) target model update where the EAGLE/MEDUSA head was not retrained — the head still infers but predicts the old target's distribution; (2) tokenizer mismatch after a vocabulary expansion; (3) numerical drift in FP8 sampling where logits at very low temperatures get quantized into a different argmax than BF16 would produce. Symptom is the same: acceptance drops to 30-40% and stays there. Alert on rolling acceptance < 50% over 1000 tokens.

Q: Does spec-decode work for diffusion-based LLMs?

Not in the standard form. Diffusion LMs (e.g., LLaDA, SEDD) generate by iterative denoising, not autoregressive decode. The "draft and verify" idea has analogues — Mercury and Inception Labs publish their own speculation-style optimizations specific to discrete diffusion — but the rejection-sampling math is different. Active research, not production-relevant in 2026 except at a handful of labs.

Q: What's the latency overhead of starting spec-decode for a new request?

First-token latency is unchanged — prefill is non-speculative. First decode token incurs one extra draft forward (~3-8 ms for a 1B draft head) before the first verification. Steady-state catches up by token 2. Net: TTFT is unaffected; inter-token latency (ITL) is dramatically lower from token 2 onward. For very-short responses (< 5 output tokens) the overhead can dominate, which is why vLLM disables spec-decode for max_tokens < 8 by default.

Q: How do I budget GPU memory for spec-decode?

For Llama-3.3-70B on H100 SXM (80 GB):

Target weights FP8: 70 GB.
EAGLE-3 draft weights BF16: 2.4 GB.
Target KV (FP8, batch 32, 8k context): ~5 GB.
Draft KV (BF16, same shape, smaller hidden dim): ~0.4 GB.
Activations + workspace: ~2 GB.

Total: ~80 GB — borderline. The practical answer in 2026 is to use H200 (141 GB) or run TP=2 across two H100s. Trying to fit Llama-3.3-70B FP8 + EAGLE-3 + batch 32 on a single 80 GB H100 is the canonical "ran out of KV" mistake.

Q: Does spec-decode change the rate-limiting factor in my deployment?

Often, yes. Before spec-decode, decode is memory-bandwidth-bound and you're paying for HBM reads. After spec-decode, decode becomes meaningfully more compute-bound (verification is denser than vanilla decode), and the binding constraint can shift to tensor-core utilization. On B200 / GB200 this matters less because both bandwidth and FLOPs scale; on H100 it matters a lot, and you may discover your bottleneck moved from "out of bandwidth" to "out of compute headroom for verification".

Q: Should I run spec-decode in the same process as the target, or in a sidecar?

Same process. The draft and target need to share GPU memory and CUDA streams. Sidecar deployments (separate draft server hit over gRPC) were tried in 2023; the network hop ate the savings. Every modern stack — vLLM, SGLang, TRT-LLM, LMDeploy — runs draft and target in one process with overlapped streams.

Q: How does spec-decode interact with LoRA hot-swapping?

Each LoRA changes the target distribution, which changes what the draft needs to predict. Three strategies in production: (1) train an EAGLE head per LoRA — best acceptance, painful operations; (2) train one EAGLE head on a mix of LoRA outputs — slight acceptance penalty (5-8 points) but operationally simple; (3) train EAGLE on base model and accept that LoRA-specific acceptance suffers — easiest, can drop acceptance by 10-15 points for divergent LoRAs. For the multi-LoRA story specifically, see the multi-tenant LoRA serving guide.

Q: What does the rejection-sampling math look like for top-p / top-k sampling?

Standard speculative sampling assumes you sample from the full distribution. For top-p (nucleus) or top-k, you have to apply the same truncation to both target and draft before computing the rejection ratio. The naive implementation that truncates only the target inflates acceptance artificially and produces samples that are technically not from the target distribution. Correct implementations (vLLM, SGLang) truncate symmetrically. The bug is common in toy implementations.

Q: Does spec-decode interact with reasoning-model "thinking" tokens?

Yes, and not always favorably. Thinking phases in models like o1, R1, and DeepSeek-R1 generate exploratory chains-of-thought with high token-level entropy. Acceptance during thinking is typically 45-55%, vs. 70-80% during the final-answer phase. Most serving stacks treat these uniformly. The optimization opportunity is to detect entry into the answer phase (typically after </think> or equivalent) and bump K from 3 to 6. SGLang exposes this as a token-pattern hint. See the reasoning model serving guide.

Q: Can I A/B test spec-decode variants safely?

Yes, because the output distribution is provably identical to the no-spec baseline. The right A/B compares latency and throughput on identical traffic. The wrong A/B compares "quality" by sampling outputs from each arm — outputs differ because sampling is stochastic, not because spec-decode is biased. Use deterministic prompts with T=0 for any output-equivalence check, and statistical metrics (KL divergence, perplexity match) on stochastic sampling at scale.

Q: What's the cost of getting spec-decode wrong?

The provable failure is the rejection ratio computed incorrectly — accepted tokens drift away from the target distribution, model behavior subtly degrades, no test catches it because individual outputs look fine. The 2024 SGLang bug where top-p truncation was applied only to the target took two weeks to detect; production users reported "slightly worse" outputs. Always validate against a vanilla-decode oracle on a held-out eval suite when changing spec-decode code paths.

Speculative decoding research and emerging variants

The space continues to evolve. What to watch.

Tree drafting innovations

Beyond EAGLE-2's dynamic tree, recent papers explore:

Specinfer: static-shape trees with multiple draft models.
Sequoia: hardware-aware tree shape optimization.
Recursive drafting: drafts of drafts.

Each tweaks the tradeoff between exploration and verification cost.

Cross-request speculation

Use successful drafts from one request as priors for similar requests. Active research.

Could provide 10-20% additional speedup on workloads with semantic similarity across requests.

Adaptive K

Beyond just auto-tuning K per workload, adapt K per token within a single sequence. Confident positions get high K; uncertain positions get K=1.

Quality-aware verification

Don't just verify against target distribution; verify quality criteria (e.g., output passes certain checks). Combines verification with sanity testing.

Speculative reasoning models

Specialized variants for o1/R1-style reasoning models. The thinking phase has different statistics; specialized drafts can help.

Hardware-accelerated drafting

Run drafts on a separate cheaper accelerator (CPU, smaller GPU) while target runs on H100. Frees up compute on the target.

Some 2025 research demonstrates this; production adoption is limited.

Distilled drafts

Train the draft model via distillation from the target. Better acceptance than independently-trained drafts.

Modern EAGLE training uses this implicitly.

When to skip speculative decoding entirely

Spec-decode isn't always the right answer.

Single-stream inference of small models

7B-class models on single GPUs are typically compute-bound. Spec-decode gives little speedup.

For Llama-3 8B on H100: spec-decode might give 10-20% speedup, not worth the complexity.

Very high-entropy workloads

Creative writing, brainstorming, exploration tasks with high-temperature sampling.

Acceptance rates are too low. Spec-decode break-even or slightly negative.

Memory-pressured deployments

If you're already KV-bound, the draft's extra memory shrinks concurrent requests. The throughput-per-request gain is offset.

Latency-critical with very tight TTFT

Spec-decode helps decode but not prefill (first-token latency). If your bottleneck is TTFT, focus on prefill optimization (chunked prefill, faster hardware).

When draft model isn't available

For a custom-architecture target model, you may not have a compatible draft. Training one is feasible but takes effort.

For these cases: skip spec-decode. Other optimizations (paged attention, prefix caching, FP8 KV) provide more reliable wins.

Implementation deep dive: spec-decode in vLLM

How spec-decode is actually implemented in vLLM.

Architecture

RequestQueue → Scheduler
                  ↓
                  ↓
        SpeculativeWorker
        (orchestrates draft + target)
        ↓               ↓
        Draft Engine    Target Engine

The SpeculativeWorker runs both engines in coordinated fashion.

Per-step flow

Scheduler picks active sequences for this iteration.
SpeculativeWorker invokes Draft Engine for K candidate tokens.
SpeculativeWorker invokes Target Engine to verify all K in one forward.
Speculative sampling determines accepted prefix.
Updates each sequence's KV with accepted tokens.

KV cache management

Both Draft and Target have their own KV pools. PagedAttention works the same for both.

vLLM's prefix caching applies to both target and draft caches independently.

Async execution

The Draft Engine runs in a separate CUDA stream from the Target Engine. They can overlap when memory permits.

Failure handling

If the draft fails (e.g., NaN), spec-decode falls back to standard decode for that step. No quality impact.

Implementation deep dive: spec-decode in SGLang

SGLang's approach is similar but tightly integrated with RadixAttention.

Tree drafting

SGLang's RadixAttention naturally extends to tree-structured speculation.

The radix tree's nodes can branch into multiple candidates, each tracked independently.

Token-level prefix sharing

Unlike vLLM's block-level sharing, SGLang shares at the token level. Combined with spec-decode, this means draft state can be partially reused across candidate trees.

Performance

For chat workloads with shared system prompts and EAGLE-2: SGLang typically delivers 10-15% better throughput than vLLM. The integration with RadixAttention is the differentiator.

Tuning spec-decode for production

Practical knobs and their tradeoffs.

K (num_speculative_tokens)

Default: 5.
Increase if acceptance is high (>75%).
Decrease if memory-constrained.

Draft model choice

Same model family as target: highest acceptance.
Smaller variants: faster but lower acceptance.
EAGLE heads: best quality.

For Llama-3 70B target: EAGLE head is optimal. Llama-3 8B as draft works but slightly worse acceptance.

Tree topology (EAGLE-2)

Default tree structure works for most workloads.

For very predictable workloads: deeper tree. For variable workloads: shallower with more branches at each level.

Verification batching

How many candidates verified per target call.

Auto-tuned in modern stacks. Manual tuning rarely beats auto.

When to disable

If acceptance falls below 50% sustained, spec-decode hurts. Auto-tuner should detect this and disable; verify behavior in your stack.

Real production deployments

Composite case studies.

Case 1: 100M tokens/day chat service

Llama-3 70B target with Llama-3 8B EAGLE draft.
K=5 default.
Acceptance: 68% on chat traffic.
Throughput improvement: 1.9× over no spec-decode.
Cost reduction: $80k/month vs no spec-decode.

Case 2: Code completion service

CodeLlama 70B target with EAGLE-2 head.
K=6 (code is predictable).
Acceptance: 85%.
Throughput improvement: 2.7×.
Latency improvement: ITL drops from 35ms to 13ms typical.

Case 3: Agent with structured output

Claude-3.5-style instruction-tuned model.
EAGLE with K=6.
Acceptance: 88% on structured output.
Throughput improvement: 2.6×.
Critical for agent latency.

Case 4: Creative writing platform

Llama-3 70B with EAGLE.
K=3 (high temperature, lower predictability).
Acceptance: 51%.
Throughput improvement: 1.3×.
Modest gain; barely worth integration cost.

For workloads in case 4's regime, consider not deploying spec-decode.

Spec-decode and other optimizations

Spec-decode composes with other inference optimizations.

Spec-decode + paged attention

No conflicts. KV cache is paged regardless (see the KV cache deep dive). Works straightforwardly.

Spec-decode + prefix caching

Target's KV cache uses prefix caching. Spec-decode's draft uses its own (possibly with prefix caching).

Net throughput: paged + prefix + spec all multiply. For broader serving-stack tradeoffs, see the LLM serving guide and disaggregated inference.

Spec-decode + FP8 KV

Same KV format applies to both target and draft. No interaction. See quantization tradeoffs for why FP8 KV is now common.

Spec-decode + multi-LoRA

Tricky. Each LoRA potentially needs its own draft head.

Practical compromise: shared draft head trained on diverse LoRA outputs. Acceptance may be slightly lower for specific LoRAs.

Spec-decode + chunked prefill

No interaction. Spec-decode is decode-only.

Spec-decode + speculative decoding (recursive!)

Theoretically possible. In practice, marginal gains.

The optimizations stack but with diminishing returns. For typical production: paged + prefix + FP8 KV + spec-decode is the sweet spot.

Theoretical bounds and limits

How fast can speculative decoding go? The theoretical limits.

Maximum speedup

If the draft is perfect (predicts target's next token always): K accepted per step. Speedup approaches K.

In practice: K capped by KV pressure and verification overhead. Realistic max: 4-5× speedup.

Acceptance rate ceiling

Acceptance is bounded by entropy of the target distribution. If the target's entropy is high (uncertain about next token), even a perfect draft has low acceptance.

For typical chat: entropy is moderate, ~70% acceptance ceiling at K=5.

For deterministic output (code, structured): entropy is low, 90%+ acceptance.

For creative writing: entropy is high, 50-60% acceptance ceiling.

Diminishing returns with K

Even with perfect draft, K can't grow arbitrarily because:

KV memory grows linearly with K.
Verification compute grows linearly with K.
At some point, the gains diminish.

Sweet spot: K=4-6 for most workloads.

Theoretical limit comparison

Metric	Vanilla decode	Spec decode (K=5, 70% accept)	Theoretical (K=10, 90% accept)
Tokens per step	1	~3.5	~9
Speedup	1.0×	~3.5×	~9×

Theoretical limits are aspirational. Real-world speed-up: 2-3× typical.

Spec-decode in agent workloads

Modern AI agents make heavy use of speculative decoding.

Why agents are well-suited

Agents typically:

Generate structured output (JSON tool calls, function args).
Follow predictable templates.
Reuse common patterns.

All of which favor high acceptance rates.

Typical agent metrics

For agentic workloads:

Acceptance rate: 80-90%.
K = 5-7.
Speedup: 2.5-3×.

This is the killer use case for spec-decode. See agent serving infrastructure and reasoning-model serving for adjacent patterns.

Combined with structured output

When agents output JSON or other structured formats:

Structure constrains output → highly predictable.
Speculative drafts almost always correct.
Combined with logits-level structure constraints (Outlines, SGLang): very high acceptance.

For JSON output specifically: 90%+ acceptance is common.

Latency implications

For agents, end-to-end latency matters more than throughput. Spec-decode reduces:

Time per agent step (faster decode).
Overall agent task duration.

Critical for interactive agents where users wait for completion.

Examples

ReAct-style agents generating thought + action: high acceptance.
Coding agents producing edits: very high acceptance.
Customer support agents with tools: high acceptance.

Ship spec-decode for any production agent workload.

Future of speculative decoding

Where the field is going.

Adaptive variants

Auto-tuning K, draft tree shape, even draft model selection per request. Active research.

Cross-request speculation

Use successful drafts from previous requests for similar new requests. Promising for repetitive workloads.

Hardware-accelerated drafting

Specialized chips for draft computation. Could enable more aggressive speculation.

Spec-decode for long-context

For very long generations, K can be larger (more amortization). New variants explore this.

Spec-decode for reasoning models

Reasoning models have high entropy in thinking phase. Specialized variants under development.

Convergence with quantization

Both spec-decode and quantization speed up inference. They compose; future may unify them.

Standardization

OpenAI-compatible APIs may add spec-decode metadata. Cross-stack interoperability.

The field is mature but continues to evolve.

When spec-decode breaks: edge cases

Specific situations where standard spec-decode fails.

Very low temperature (T < 0.1)

At very low temperature, target's distribution is essentially deterministic. Draft's distribution differs even slightly.

Result: very low acceptance. Spec-decode is break-even or negative.

Mitigation: detect low temperature, disable spec-decode for those requests.

Very high temperature (T > 1.5)

Distributions become flat. Draft and target both produce highly variable outputs.

Acceptance rate is roughly the dot product of distributions. Lower than expected.

Mitigation: same as above, disable for high-temperature workloads.

Adversarial inputs

If users craft prompts designed to confuse the draft: acceptance drops.

Rare in practice but possible. Mitigation: detect anomalies in acceptance rate, fall back to standard decode.

Streaming with backpressure

If client reads slowly, spec-decode's K-tokens-at-once burst pattern may overrun the client buffer.

Mitigation: throttle on backpressure. Most stacks handle this automatically.

Mid-generation parameter changes

If user mid-stream changes max_tokens or other parameters: spec-decode may need to reset.

Mitigation: detect changes, restart from current state.

Stateful operations

Tool calls, function calls that modify state: spec-decode is fine for the tokens but external state changes need careful handling.

For most stacks: tools execute after generation completes; spec-decode just speeds up generation.

Spec-decode and quality monitoring

Ensuring spec-decode doesn't silently degrade quality.

Acceptance rate as a leading indicator

If acceptance drops below 50%: investigate.

Causes:

Draft model bug.
Target model update without retraining draft.
Workload distribution shift.
Numerical issues in spec sampling.

Track acceptance rate continuously.

Output quality monitoring

Periodic comparison of spec-decode output vs vanilla output:

Same prompt to both.
Compare similarity.

Should match within 99%+ similarity (since spec-decode is exact).

If divergence: implementation bug.

Implementation tests

Unit tests for the speculative sampling formula. Verify the math.

Integration tests on known workloads. Compare to baseline.

Regression tests on every code change.

A/B testing of variants

Try EAGLE-1 vs EAGLE-2 vs MEDUSA on your workload. Pick the winner.

Don't assume one works best for everyone.

Dashboard metrics

For production monitoring:

Acceptance rate (per workload type).
K used (auto-tuned variants).
Spec-decode failures (NaN, etc.).
Throughput improvement vs vanilla baseline.

Alert on degradation in any metric.

Glossary

Acceptance rate: fraction of drafted tokens that survive verification.
Draft model: small model generating candidate tokens.
EAGLE / EAGLE-2: spec-decode using a draft head sharing target's hidden states.
K: draft length, number of candidate tokens per step.
MEDUSA: spec-decode using multiple decoding heads on the target.
REST: retrieval-based spec-decode.
Speculative sampling: rejection sampling step ensuring statistical correctness.
Target model: large model being accelerated.
Vanilla spec-decode: original variant with separate draft model.

References

Leviathan et al., Fast Inference from Transformers via Speculative Decoding, ICML 2023. arXiv:2211.17192.
Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, 2023. arXiv:2302.01318.
Miao et al., SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification, 2023. arXiv:2305.09781.
Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, 2024. arXiv:2401.15077.
Li et al., EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, 2024. arXiv:2406.16858.
Cai et al., MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024. arXiv:2401.10774.
Fu et al., Lookahead Decoding: A Decoding Algorithm for Faster LLM Inference, 2024. arXiv:2402.02057.
Elhoushi et al., LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding, 2024. arXiv:2404.16710.
He et al., REST: Retrieval-Based Speculative Decoding, 2024.
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM), 2023. arXiv:2309.06180.
Zheng et al., Efficient Programming of Large Language Models using SGLang, 2023. arXiv:2312.07104.

Speculative decoding deployment patterns

How real production systems deploy speculative decoding.

Pattern 1: Baseline (no spec decoding)

Start without speculative decoding. Establish baseline metrics:

Throughput (tokens/sec).
Latency (p50, p99).
Quality (eval scores).
Cost per token.

This is the comparison baseline.

Pattern 2: Vanilla draft model

Add a draft model (~10x smaller than target):

Llama-3 70B target → Llama-3 8B draft.
Tune draft length (start with 4-8 tokens).
Validate quality unchanged.

Expected speedup: 1.5-2.5x.

Pattern 3: EAGLE-style speculation

Replace draft model with EAGLE:

Train EAGLE on top of base model.
Lower memory than separate draft model.
Higher acceptance rate.

Expected speedup: 2-3x with same memory footprint.

Pattern 4: EAGLE-2 with dynamic draft

EAGLE-2 dynamically adjusts draft tree:

Better acceptance rate.
Adapts to context.
Slightly more complex to deploy.

Expected speedup: 2.5-3.5x.

Pattern 5: MEDUSA (multi-head)

MEDUSA modifies the model itself:

Adds prediction heads.
No separate draft model.
Single model with built-in speculation.

Expected speedup: 2-2.5x.

Pattern 6: Hybrid

Different speculation methods for different request types:

Simple completions: vanilla draft.
Complex queries: EAGLE-2.
Code generation: longer draft.

Optimizes per workload.

Pattern 7: Speculation off

For some workloads, turn it off:

Very short responses (< 10 tokens): overhead > benefit.
Highly variable contexts: low acceptance rates.
Latency-critical and consistent: off can be more predictable.

Don't force speculation everywhere.

Choosing a pattern

Decision factors:

Memory budget.
Quality tolerance.
Workload characteristics.
Engineering complexity.

For most teams: start with EAGLE or EAGLE-2.

Speculative decoding optimization techniques

Beyond basic tuning, these techniques squeeze more performance.

Adaptive draft length

Instead of fixed draft length, adapt per request:

Track recent acceptance rate.
Increase length when accepting.
Decrease when rejecting.

Improves average throughput.

Tree-structured speculation

Speculate multiple branches simultaneously:

Better acceptance rate (pick longest accepted).
More parallel work.
Higher peak throughput.

Used by EAGLE-2 and MEDUSA.

Speculation cache

Cache speculation outputs by prefix:

For repeated queries, instant draft.
Significant for high-cache-hit workloads.

Implementation: prefix-keyed cache of accepted speculation paths.

Prefix-aware speculation

Use prefix to inform speculation:

Code completions can use code-specific patterns.
Chat can use conversation patterns.

Domain-specific tuning.

Speculation skipping

Skip speculation for low-confidence regions:

Detect via target model entropy.
Don't waste cycles on hard tokens.

Marginal but real improvement.

Multi-target speculation

Speculate for multiple target models with one draft:

Useful when serving model variants.
Single draft model amortizes across requests.

Operational complexity but better hardware utilization.

Hardware-aware optimization

For different GPU types:

H100: standard speculation works well.
B200: higher memory bandwidth changes tradeoffs.
L40S: different sweet spot for draft size.

Tune per hardware.

Quantization of draft model

Draft model can be more aggressively quantized:

INT4 draft model with FP8 target.
Frees memory for more KV cache.

Common pattern for memory-constrained deployments.

Compilation

torch.compile or TensorRT-LLM compilation:

Optimizes both target and draft kernels.
Significant speedup for steady-state inference.

For production: usually worth the complexity.

Profiling and tuning

Continuous profiling:

Acceptance rate over time.
Per-token latency.
Memory utilization.

Tuning is workload-dependent.

Speculative decoding observability

How to monitor and debug speculative decoding in production.

Key metrics

Acceptance rate: tokens accepted / tokens proposed.
Effective speedup: target tokens generated per target model forward pass.
Draft model latency: time per draft forward.
Target model latency: time per target forward.
End-to-end token latency: user-facing.

Acceptance rate breakdown

By:

Request type (code, prose, etc.).
Position in sequence (early vs late).
Draft length used.
Time of day.

Patterns suggest tuning opportunities.

Common observability failures

No metrics: flying blind.
Aggregate-only metrics: masks per-request issues.
No alerting: regressions go undetected.
Lack of A/B comparison: can't measure speculation benefit.
No quality tracking: speedup at cost of quality.

Alerting thresholds

Acceptance rate < 50%: investigate.
Speedup < 1.5x: speculation may not be worth it.
Quality regression: rollback.

Debugging tools

Per-request acceptance traces.
Distribution histograms.
Comparison logs (with vs without speculation).

For complex deployments: dedicated observability.

Speculative decoding research frontiers

What's emerging in 2026.

Self-speculation

Models that speculate using their own internal layers:

No separate draft.
Activation reuse.
Lower memory.

Active research. Some promising results.

Continuous speculation

Speculation interleaved with target generation:

Smoother throughput.
Better latency consistency.

Implementation complex but improvements showing.

LLM-aware speculation

Speculation that uses semantic understanding:

Predict syntactic structures.
Domain-specific patterns.
Combined with retrieval.

Bridges speculative decoding with broader inference optimization.

Speculative streaming

For streaming responses:

Speculate ahead of UI rendering.
Batch user-visible tokens.

User experience improvements.

Multi-step speculation

Each draft step itself speculates:

Tree of speculations.
Higher peak speedup.
More memory.

Active area.

Verification optimization

Verifying speculations is the bottleneck:

Parallel verification.
Approximate verification with quality bounds.
Hardware acceleration.

Several papers advancing.

Speculation for non-LLM workloads

Applying speculation to:

Diffusion models.
Multimodal generation.
Reasoning chains.

Active research with mixed results.

Speculation in different inference engines

How each major inference engine handles speculation.

vLLM

Native speculative decoding support:

Vanilla draft model.
EAGLE (community contributions).
N-gram speculation.
MLP speculator.

Configuration via SpecDecodeConfig. Generally well-tested.

Integration: spec decoding works alongside continuous batching, but acceptance rates can vary across batched requests.

Limitation: not all variants production-ready. EAGLE-2 still emerging.

SGLang

Strong speculative decoding:

EAGLE family well-supported.
RadixAttention helps with draft KV reuse.
Tree-structured speculation.

Often performs better than vLLM for spec decoding workloads.

TensorRT-LLM

NVIDIA's optimized inference:

MEDUSA support.
EAGLE support.
Best raw performance.

Steeper learning curve but highest throughput.

TGI (Text Generation Inference)

HuggingFace's serving framework:

Draft model speculation.
Limited variant support.

Good for simple deployments.

Ollama / llama.cpp

Lightweight local inference:

Speculation support emerging.
Optimized for laptop / single-GPU.

For local: speculation can help but less critical.

Custom inference

Building your own:

Reference implementations available.
Significant engineering effort.
Only justify for large-scale deployments.

For most teams: use existing engines.

Engine selection

For most production:

vLLM if you're already there.
SGLang for cutting-edge speculation.
TRT-LLM for absolute best performance.

Migration between engines is non-trivial.

Speculation tradeoffs in detail

Comprehensive view of when speculation helps or hurts.

Memory tradeoffs

Speculation requires:

Draft model in memory (vanilla).
Or speculative heads (MEDUSA).
Or auxiliary network (EAGLE).

Memory cost: 5-30% of target model.

Impact on KV cache:

Less memory for KV → smaller batch sizes.
Trade between speculation speedup and concurrency.

Latency tradeoffs

When speculation wins:

Memory-bandwidth-limited workloads.
Sufficient acceptance rate (>50%).
Predictable patterns.

When speculation loses:

Compute-limited workloads.
Low acceptance rate.
High variance contexts.

For latency-critical systems: measure carefully.

Throughput tradeoffs

Throughput improvements depend on:

Batch size.
Acceptance rate.
Draft model overhead.

In high-throughput batched inference: speculation often less helpful (already amortizing target compute). In single-stream low-batch: speculation more helpful.

Quality tradeoffs

Mathematically: speculative decoding is exact (matches target distribution).

Practically:

Implementation bugs can cause subtle quality issues.
Numerical precision can matter.

Validate quality on your evals.

Cost tradeoffs

Cost = (target compute + draft compute) / accepted tokens.

Cost analysis:

High acceptance rate: speculation reduces cost.
Low acceptance rate: speculation increases cost.

Typical break-even: ~50% acceptance rate.

Engineering tradeoffs

Speculation adds:

Complexity.
Failure modes.
Operational burden.

Worth it for:

High-traffic services.
Latency-critical applications.
Memory-bandwidth-limited workloads.

Not worth it for:

Low-traffic services.
Quality-critical applications without thorough validation.
Already-throughput-bound systems.

A/B testing in production

Standard approach:

Deploy with speculation toggleable.
A/B test traffic between speculative and non-speculative.
Measure latency, throughput, quality, cost.
Tune or roll back based on results.

Don't deploy speculation without measuring.

Speculation for specific workloads

How speculation performs for different use cases.

Code completion

Patterns:

Highly predictable patterns (syntax, imports).
High acceptance rate (60-80%).
Excellent fit for speculation.

Speedup: typically 2-3x.

Tuning: longer drafts work well.

Chat completion

Patterns:

Variable patterns.
Moderate acceptance (40-60%).
Good fit for speculation.

Speedup: 1.5-2.5x.

Long-form writing

Patterns:

Variable patterns.
Acceptance varies (30-70%).
Moderate fit.

Speedup: 1.5-2x.

Reasoning / chain-of-thought

Patterns:

High variability.
Lower acceptance (30-50%).
Mixed fit.

Speedup: 1.5x or less.

Translation

Patterns:

Structured output.
High acceptance.
Good fit.

Speedup: 2-3x.

RAG / retrieval-augmented generation

Patterns:

Heavily depends on retrieved context.
Variable acceptance.
Mixed fit.

Speedup varies.

Multimodal generation

Patterns:

Different distribution per modality.
Speculation less mature.
Limited speedup.

Status: emerging.

Selection guidance

For any workload:

Measure acceptance rate.
Calculate effective speedup.
Compare to throughput requirements.
Decide based on data.

Don't assume — measure.

Speculation production playbook

Step-by-step playbook for deploying speculation in production.

Step 1: Establish baseline

Without speculation:

Measure throughput.
Measure latency.
Measure quality.
Measure cost.

This is your reference.

Step 2: Choose initial method

Start simple:

Vanilla draft model (smaller version of target).
Or EAGLE if available for your model.
Avoid more exotic methods initially.

Step 3: Implement and test

Implementation in your inference engine:

Configure appropriately.
Test on representative traffic.
Compare to baseline.

Step 4: Tune draft length

Sweep draft lengths (4, 6, 8, 12 tokens):

Find peak throughput.
Validate quality unchanged.
Note acceptance rates.

Step 5: Deploy to production gradually

10% traffic A/B test.
Monitor closely.
Roll forward if metrics good.

Step 6: Monitor continuously

Acceptance rate.
Latency distribution.
Throughput.
Quality eval at intervals.

Step 7: Tune further

Based on production data:

Adjust draft length.
Try other methods.
Improve based on workload patterns.

Step 8: Handle edge cases

Long contexts.
Specific request types.
Failure modes.

Each may need special handling.

Step 9: Document and share

Document your speculation setup:

Method used.
Configuration.
Performance numbers.
Known issues.

Institutional knowledge.

Step 10: Continuous improvement

Speculation evolves:

New methods emerge.
Hardware changes.
Workloads change.

Re-evaluate periodically.

Speculation interplay with continuous batching

Speculation interacts with continuous batching in subtle ways.

Continuous batching basics

Modern inference engines pack multiple requests in a batch. New requests join, finished ones leave.

This dynamic batching maximizes GPU utilization.

Speculation in batch

When some batch requests use speculation:

Different requests at different positions.
Speculation must coexist with diverse states.

vLLM and SGLang handle this — but tuning matters.

Memory implications

Speculation memory + batch state:

Total memory budget tight.
Trade between speculation and batch size.

Acceptance rate variance

In a batch:

Different requests have different acceptance rates.
Aggregate metrics mask variance.

Profile per-request when debugging.

Optimal configuration

For batched workloads:

Smaller draft length than single-stream.
Adaptive draft length helps.
Memory budget tight.

Common issues

OOM from over-aggressive speculation.
Throughput variance from heterogeneous acceptance.
Latency tail issues.

Test thoroughly.

Best practice

Run with continuous batching enabled, speculation enabled, validate aggregate and per-request metrics.

This is how production should look.

Speculation in serverless inference

Speculation in serverless / pay-per-request inference.

Provider perspective

For serverless inference providers:

Speculation increases throughput per GPU.
Higher GPU utilization.
Better margin per request.

Most major providers use speculation behind the scenes.

Customer perspective

For customers:

Lower latency in some cases.
Same quality.
Same price (usually).

Generally invisible win.

Provider implementation

Providers handle:

Choice of speculation method.
Tuning per workload.
Quality validation.
Failure handling.

Customer doesn't usually see details.

Pricing implications

If providers charge by token:

Speculation doesn't change tokens charged.
Provider keeps efficiency gain.

If providers charge by time / GPU-second:

Speculation could lower customer cost.

Most modern providers charge by token.

What customers should know

Speculation may be active.
Quality should match non-speculation.
Latency should be no worse.

Verify these.

Customer-facing speculation

Some providers expose speculation as option:

Custom draft models.
Tunable parameters.
Premium tier.

Most don't.

Future trends

Speculation in serverless:

Becoming default.
More sophisticated methods.
Better customer transparency.

This is evolving rapidly.

Speculation summary and recommendations

Wrapping up.

The bottom line

Speculative decoding is one of the most effective inference optimizations available today, providing 2-3x speedup with minimal quality impact.

Recommendation by deployment

Production LLM serving (high traffic): implement speculation. Use EAGLE or vanilla.
Production LLM serving (low traffic): skip unless single-stream latency critical.
Research / experimentation: skip — quality validation overhead.
Edge / on-device: usually skip — memory overhead.
Multi-tenant SaaS: yes — throughput gains directly translate to capacity.

Tools recommendation

vLLM: reasonable speculation support, easiest to deploy.
SGLang: best-in-class speculation support.
TRT-LLM: best raw performance, more complex.

Future direction

Speculation will continue to improve. Watch for:

Better acceptance rates from research.
Hardware support for speculation.
Integration with other optimizations.

Final advice

Don't speculate (in the bad sense). Measure carefully. A/B test. Validate quality.

Speculation should make your service better — not worse. The math is on your side, but implementation details matter.

Speculation FAQ extension

Speculation cost analysis

Cost analysis of speculation.

Compute cost

Cost = target compute + draft compute.

Per accepted token:

Without spec: 1 target forward.
With spec: (1 target + k draft) / accepted_tokens.

Break-even analysis:

For acceptance rate p, k tokens drafted: cost reduction if (k * draft_cost + 1 target) / E[accepted] < target_cost.

Generally: speculation reduces cost when acceptance > ~30%.

Memory cost

Speculation requires:

Draft model in GPU memory.
Or speculative heads.
Speculation buffers.

Typically 5-20% extra memory.

Engineering cost

One-time:

Implementation.
Testing.
Validation.

Ongoing:

Monitoring.
Tuning.
Updates.

Operational cost

Additional metrics to track.
More complex debugging.
Failure mode handling.

When cost-benefit is positive

For most production:

Throughput gain > engineering cost.
Latency improvement valuable.
Memory available.

When it's negative:

Low traffic (engineering cost not amortized).
Very high quality requirements (validation cost high).
Memory-constrained.

Long-term value

Speculation tends to:

Improve over time (better methods, better tools).
Decrease in cost.
Become more standard.

Investment now pays dividends.

Speculation in different deployment scales

How speculation behaves at different scales.

Single-user, low-traffic

Pattern:

One inference instance.
Sporadic requests.

Speculation impact:

High benefit per request.
Memory cost matters less.
Engineering complexity high relative to benefit.

Recommendation: skip unless single-request latency is critical.

Multi-user, moderate traffic

Pattern:

Several inference instances.
Steady traffic.

Speculation impact:

Moderate benefit.
Memory cost can affect batch size.
Worth investigating.

Recommendation: A/B test, measure carefully.

High-traffic, batch-optimized

Pattern:

Many GPUs.
High throughput goals.
Continuous batching.

Speculation impact:

Throughput gain depends on workload.
Often less impressive than single-stream gains.
Memory matters a lot.

Recommendation: investigate per-workload.

Latency-critical

Pattern:

p99 latency goals.
Variable load.

Speculation impact:

Helps mean latency.
May hurt p99 (variance).

Recommendation: tune for p99, may use selectively.

Edge / on-device

Pattern:

Limited memory.
Battery constraints.

Speculation impact:

Memory cost prohibitive often.
Energy cost may not be worth.

Recommendation: typically skip.

Hybrid serving

For workloads with mixed characteristics:

Route based on request type.
Speculation on/off per route.

Operational complexity but optimal.

Speculation common pitfalls

What to avoid.

Pitfall 1: Skipping quality validation

Just because tests pass doesn't mean output is identical. Validate.

Pitfall 2: Wrong draft model size

Too small: low acceptance rate. Too large: high overhead. Sweet spot is ~10-15x smaller.

Pitfall 3: Static draft length

Different requests benefit from different lengths. Adaptive is better.

Pitfall 4: Ignoring memory cost

Draft model uses memory. Measure impact on max batch size.

Pitfall 5: Insufficient testing

Speculation is complex. Test thoroughly before production.

Pitfall 6: Optimizing for wrong metric

Throughput vs latency tradeoff. Optimize what matters for your service.

Pitfall 7: Not handling failures

Speculation can fail in subtle ways. Plan for fallback.

Pitfall 8: Incompatible with quantization

Some speculation methods don't compose well with aggressive quantization. Test combinations.

Pitfall 9: Not measuring end-to-end

Microbenchmarks can mislead. Measure user-visible latency.

Pitfall 10: Premature optimization

For low-traffic services: speculation may not be worth it.

Speculation theory deep dive

The mathematical foundations.

Why speculative decoding is exact

The acceptance/rejection scheme ensures:

For each position, the accepted token is sampled from the target distribution.
Even though we used a draft proposal.

Proof sketch:

Acceptance probability is min(1, p_target / p_draft).
This rejection sampling exactly samples from p_target.

So speculation produces tokens identical to baseline (in distribution).

Acceptance rate analysis

Acceptance rate depends on:

KL divergence between draft and target.
Lower KL → higher acceptance.

For draft models that mimic target well: high acceptance. For draft models that differ: low acceptance.

Theoretical speedup

Maximum speedup with k-token speculation is k+1 (target call processes k+1 positions).

Effective speedup is:

E[accepted tokens] + 1.
Where E[accepted] depends on acceptance rate.

For acceptance rate p, expected accepted tokens = (1 - p^k) / (1 - p) - 1.

For p=0.7, k=8: ~3.5 expected accepted tokens. Effective speedup ~4.5.

Optimal draft length

Optimization: maximize speedup minus overhead.

Speedup grows but plateaus with k (eventually all rejection). Overhead grows linearly with k (draft model cost).

Optimal k satisfies: marginal speedup = marginal overhead.

In practice: empirically determined.

Tree-structured speculation theory

Instead of single chain of k tokens, branch into tree:

More candidates → higher acceptance.
Larger trees → more compute.

EAGLE-2 uses tree speculation with adaptive structure.

Information theory perspective

Speculation extracts information from target's predictability:

Predictable text → high acceptance.
Surprising text → low acceptance.

Speculation exploits language's redundancy.

Limit analysis

Theoretical maximum speedup:

Bounded by hardware constraints.
Bounded by inherent text predictability.

Practically: 3-5x is the realistic ceiling for most workloads.

EAGLE-3 in detail

EAGLE-3 (mid-2024 paper) is the production-relevant successor to EAGLE-2. The headline change: training-time data augmentation removes the dependency on the target model's exact features, raising acceptance rates and stabilising the speedup curve. The architectural moves that matter in practice:

Multi-layer feature aggregation: EAGLE-3 uses features from multiple layers of the target (not just the last hidden state), which captures more of the target's internal distribution. Acceptance rate rises by approximately 5–10 percentage points on Llama-class targets across our reading of published numbers.
Training-time test loss: EAGLE-3 introduces an explicit "test loss" during draft training that mirrors inference-time tree construction. This narrows the train-test gap that caused EAGLE-2 acceptance to drop on out-of-distribution prompts.
Removal of "feature copy" dependency: EAGLE-2's draft fed on copies of the target's last-layer feature; in long sequences this caused drift. EAGLE-3 trains the draft to predict the next features and tokens jointly, making the draft self-sufficient.

Practical implication: EAGLE-3 is roughly a 10–20% additional speedup on top of EAGLE-2 in published benchmarks, with similar memory cost. For new deployments, EAGLE-3 is the right starting point; for existing EAGLE-2 deployments, the switch is usually worth the engineering effort if you control draft training.

EAGLE-3 vs EAGLE-2 vs MEDUSA-2 vs Lookahead

Aspect	EAGLE-2	EAGLE-3	MEDUSA-2	Lookahead
Acceptance rate (chat, Llama 70B)	~0.7	~0.78	~0.55	~0.45
Speedup (chat)	2.5–3x	2.8–3.4x	1.8–2.4x	1.5–2x
Memory overhead	~5–10%	~5–10%	~3–6%	minimal
Training requirement	Draft training	Draft training	Self-distillation	None
Hot-swap difficulty	Medium	Medium	Low	Lowest
Stack support (vLLM, SGLang, TRT-LLM)	Native	Adding	Native	Native

Numbers are approximate and vary by workload.

MEDUSA-2 and self-distillation

MEDUSA-2 (mid-2024) is the operational successor to the original MEDUSA. The core insight: instead of training extra heads from scratch, distil the target into itself with multi-token prediction heads. The procedure:

Take the production target model (e.g., Llama 70B).
Generate synthetic training data using the target itself.
Train K additional heads (typically 4–8) that predict tokens K+1, K+2, ... K+N given the same hidden state used for token K+1.
The base model is fine-tuned alongside the heads to slightly improve calibration.
At inference, the heads propose multiple candidates per step; verification selects accepted prefixes.

The benefit over MEDUSA-1: higher acceptance rates because the heads are trained with the actual target distribution, not a generic corpus. The cost: requires fine-tuning the target model. For teams that control the model, MEDUSA-2 is operationally clean; for teams using third-party weights, EAGLE-3 is easier because it doesn't touch the target.

Lookahead Decoding and Jacobi iteration

Lookahead Decoding (Fu et al., 2024) doesn't use a draft model at all. It exploits the target's ability to predict multiple tokens at once if you give it a guess.

The mechanics:

Jacobi iteration: solve a fixed-point problem y = f(y) by iterating. For language modelling, this means: guess a sequence of K tokens, run the target once to get K refined guesses, repeat until convergence.
N-gram cache: keep a cache of recently-seen N-grams; when the target produces a token, check the N-gram cache for likely continuations.
Lookahead branches: at each step, the target evaluates the current position plus several lookahead branches in parallel.

The output is identical to vanilla decoding (provably). The speedup comes from amortising the target's HBM cost across multiple positions per step. Typical speedup: 1.5–2x on chat, less on highly creative content.

Strengths: zero new infrastructure, no draft model, no training. Weaknesses: weaker speedup than EAGLE; tuning the lookahead window and N-gram cache size is workload-dependent.

vLLM exposes Lookahead via --speculative-method ngram with the equivalent settings. SGLang's "Eagle" path can fall back to Lookahead when no draft model is available.

REST: retrieval-based speculative decoding

REST (He et al., 2024) replaces the draft model with a datastore lookup. The intuition: many decoded tokens follow predictable patterns; if a similar context has been decoded before, the continuation is likely similar.

Architecture:

Datastore: a corpus of (context, continuation) pairs indexed by the embedding of the context.
At decode time: embed the current context; nearest-neighbour search returns several likely continuations; verify with the target.
No draft model required.

REST's strengths: zero training, easy to update (just add new pairs to the datastore), good for tasks with repeated patterns (code, structured outputs, FAQ-style chat). Weaknesses: weaker on creative content where patterns don't repeat; datastore size and quality matter; embedding-based retrieval adds latency.

Production status: less common than EAGLE in 2026 but useful for specific high-repetition workloads.

BiLD: Big-Little Decoder

BiLD (Big-Little Decoder, Kim et al., 2023) introduced a control structure that pairs a small "little" model with a large "big" model in a more flexible way than vanilla speculative decoding:

The "little" model decodes tokens until a fallback policy triggers (uncertainty threshold, token boundary heuristics).
The "big" model is invoked to verify and continue.
Unlike strict speculative decoding, BiLD can in some configurations sacrifice exact distributional equivalence for additional speedup.

BiLD's main contribution to the field was establishing the design space — most subsequent variants (EAGLE, Medusa, Lookahead) chose the distribution-preserving path BiLD did not commit to.

SpecInfer and tree-structured verification

SpecInfer (Miao et al., 2023) introduced tree-structured verification: instead of verifying a linear sequence of K draft tokens, verify a tree of candidate continuations. The tree captures more branches per target call, raising the expected number of accepted tokens per verification.

Tree decoding is the basis for EAGLE-2's tree variant and SGLang's RadixAttention-aware speculative path. The trade-off: trees use more memory and compute per verification call, but each call accepts more tokens. The math typically favours moderate-depth, moderate-width trees (e.g., 4 wide × 6 deep) over linear sequences.

PASS: pipeline-parallel speculative decoding

PASS (Pipeline-parallel Architecture for Speculative Sampling) is the design pattern that makes speculative decoding work well in multi-GPU pipeline-parallel serving. The challenge: in a vanilla pipeline-parallel deployment, the draft model and the target model may live on different GPU groups; coordinating drafts across pipeline stages adds latency.

PASS solves this by:

Placing the draft on the same pipeline stage as the first decoder layer of the target.
Streaming drafts forward through the pipeline.
Using async communication primitives (NCCL all-gather and reduce-scatter) to overlap draft generation with target verification.

Production deployments at frontier labs use PASS-like patterns; the implementation is bespoke to each lab's serving stack.

Per-stack support deep dive

The actual configuration surface for speculative decoding across major serving stacks as of mid-2026.

vLLM

vLLM has the most mature speculative-decoding support among open-source stacks. Configuration (from CLI flags):

--speculative-model <name>: EAGLE, Medusa, ngram, or a draft model checkpoint.
--num-speculative-tokens N: the K parameter.
--speculative-draft-tensor-parallel-size: separate TP for the draft.
--speculative-disable-by-batch-size: turn off speculation above a batch threshold.
v1 of vLLM (rolling out through 2025–2026) adds first-class tree decoding and dynamic K selection.

EAGLE and EAGLE-2 are first-class; EAGLE-3 support is rolling out. Medusa is supported with self-distilled weights. Lookahead is supported via the ngram path.

SGLang

SGLang's speculative-decoding path is integrated with RadixAttention, which adds prefix-cache awareness to drafts. Configuration via --speculative-algorithm eagle and friends. SGLang generally targets the same workloads as vLLM with a different architecture (RadixAttention-first).

TensorRT-LLM

TRT-LLM (NVIDIA) supports speculative decoding via its plugin architecture. Configuration is more verbose (requires building engine files with speculative-enabled plugins). EAGLE, Medusa, and lookahead are supported. TRT-LLM's strength is on NVIDIA hardware-specific optimisations; the trade-off is engineering effort.

llama.cpp

llama.cpp added speculative decoding (draft model path) for CPU and Apple Silicon. The -md (draft model) flag specifies the draft; default K is small. Speedup is meaningful for chat workloads on M-series Macs.

LMDeploy

LMDeploy (InternLM) supports speculative decoding with Medusa-style heads. Aligned with the InternLM model family.

TGI

Text Generation Inference (HuggingFace) supports basic speculative decoding via draft model; less mature than vLLM/SGLang/TRT-LLM.

Comparison table

Stack	EAGLE-2	EAGLE-3	MEDUSA	Lookahead	Tree decoding	Dynamic K
vLLM (v1)	Yes	Adding	Yes	Yes (ngram)	Yes	Yes
SGLang	Yes	Adding	Yes	Yes	Yes	Yes
TRT-LLM	Yes	Yes	Yes	Yes	Yes	Limited
llama.cpp	Via draft	No	No	No	No	No
LMDeploy	Limited	No	Yes	Limited	Limited	Limited
TGI	Via draft	No	Limited	No	No	No

Chunked prefill interaction

Chunked prefill splits long prefill into chunks that fit within the same batch as decode requests. With speculative decoding, the interaction is:

During prefill chunks, the draft model is idle.
When prefill completes and decode begins, the draft is engaged.
The draft's KV cache must be populated for the prefill content before drafting can start — this is the "warm-up" cost.

In practice, chunked prefill and speculative decoding compose cleanly in vLLM and SGLang. The serving optimisation is to schedule speculative-decoding requests to maximise GPU utilisation during decode while accommodating prefill traffic.

Disaggregated prefill/decode interaction

In disaggregated serving (separate GPU pools for prefill and decode), speculative decoding lives in the decode pool. The interaction:

Prefill servers produce KV caches for the target.
Decode servers run target + draft (or target + EAGLE head).
KV cache transfer between pools happens before decode begins; the draft's KV is built locally on the decode side.

For more on disaggregated serving see disaggregated inference. Speculative decoding is a decode-side optimisation; it composes with disaggregation but doesn't require it.

Mixture-of-experts target with speculative decoding

MoE targets (Mixtral, DeepSeek-V3, GPT-4-class) have higher per-token compute variance than dense models because the active experts change per token. Implications for speculative decoding:

Draft model strategy: a dense small draft is simplest. An MoE draft is rarely worthwhile (extra complexity, modest accept-rate gain).
EAGLE head: works fine on MoE targets; the head shares the target's hidden state.
Acceptance rate: typically similar to dense targets if the draft is dense and well-trained.
Throughput math: MoE targets are already throughput-friendly because active experts is smaller than total experts; the marginal benefit of speculative decoding is similar in proportion but smaller in absolute terms.

Reasoning models and speculative decoding

Reasoning models (DeepSeek R1, OpenAI o-series, Claude 4.5/4.6 thinking) generate substantially more tokens than non-reasoning models. The "thinking tokens" portion is typically more predictable than final-output tokens (it follows reasoning chains that the draft can learn).

Empirical pattern: acceptance rates on thinking tokens are typically higher than on output tokens, so speculative decoding's speedup is larger overall on reasoning workloads — sometimes 3x or more. Caveats:

Some reasoning patterns are highly idiosyncratic (the model "discovers" a new chain of reasoning that the draft hasn't seen).
Reasoning model verifiers (e.g., self-consistency checking) interact non-trivially with speculative decoding.

Production reasoning-model deployments increasingly use EAGLE-3 or comparable to amortise the longer output cost.

Structured output + speculative decoding

Structured-output decoding (constrained generation for JSON, code, function calls) interacts with speculative decoding through the constraint mechanism:

The constraint mask (which tokens are legal at each position) is applied during target verification.
If the draft proposed a token that's illegal under the constraint, it's rejected.
High-constraint contexts (strict JSON schema) can have lower acceptance rates because the draft's distribution doesn't match the constrained distribution.

Pragmatic approach: train the draft with structured-output examples (so it learns to propose schema-valid tokens) or accept the slight acceptance-rate degradation for the simpler implementation.

Multimodal targets

Vision-language models (Llama 3.2 Vision, GPT-4V, Claude Vision) accept images and produce text. The vision tokens are processed in prefill; the speculative-decoding logic applies to text-token generation, similar to text-only targets. Drafts for VLMs are typically text-only (the small draft doesn't process images); acceptance rates are comparable to text-only targets once vision context is established.

CPU and edge speculative decoding

For CPU and small-GPU edge deployments (M-series Macs, Jetson, mobile), speculative decoding's value is real but tuning differs:

Memory budget is tight: a 1B draft for a 8B target is acceptable; a 7B draft for a 70B target is not.
Latency vs throughput: edge deployments are usually latency-first; speculative decoding helps because it reduces tokens-per-second-per-user latency.
Power: speculative decoding doesn't increase total work; it amortises work across fewer GPU/CPU clock cycles. On laptops, this can reduce power consumption.
llama.cpp: the primary stack for edge speculative decoding; the draft model path is mature.

Acceptance rate by position-in-sequence

Acceptance rates are not uniform across positions in a generated sequence. Patterns:

Early positions (right after prefill): often higher acceptance because the context is clear and the draft has plenty of signal.
Middle positions: stable acceptance rate.
Late positions: can drop if the model is "discovering" novel content (long-form generation, creative tasks).
Boundary positions (sentence ends, code-block transitions): often lower acceptance because the model branches.

Dynamic K adapts to this: high K when acceptance is high, low K when boundaries approach. Modern stacks (vLLM v1, SGLang) implement adaptive K based on recent acceptance history.

FP8 and quantised drafts

Quantising the draft model is one of the cheapest speedup additions to a speculative-decoding setup:

FP8 draft: typically little accuracy loss; near-2x draft throughput on H100/H200.
INT4 draft (AWQ, GPTQ): 4–6x draft throughput but more acceptance-rate loss.
FP4 draft (Blackwell): emerging; minimal published benchmarks.

The target model remains at higher precision (BF16, FP8). The acceptance-rate cost of a quantised draft is usually small (1–3 percentage points) and the throughput gain is meaningful. For high-throughput deployments, an FP8 draft + BF16 target is a common production combo.

Failure modes

The ways speculative decoding fails in production:

Draft drift: the draft was trained on different data than the target's current distribution. Acceptance rate crashes. Fix: retrain the draft.
Target throughput cap: at very high concurrency, the target's verification path is already saturated; speculative decoding adds overhead without speedup. Fix: disable speculation above a threshold (vLLM's --speculative-disable-by-batch-size).
Workload mismatch: highly creative or low-predictability content has low acceptance; speedup is marginal. Fix: route low-acceptance traffic to non-speculative path.
Constraint conflict: structured-output constraints + speculative drafts can produce poor acceptance. Fix: train draft with structured examples or accept degradation.
KV cache pressure: the draft's KV doubles part of the memory footprint. Fix: smaller draft, or share KV via EAGLE-style architecture.
MoE expert drift: in MoE targets, expert-routing patterns differ between draft and target. Fix: align routing or use dense draft.
Speculation deadlock in batched serving: speculative paths can deadlock if not handled carefully in continuous batching. Fix: rely on mature stacks; don't roll your own scheduling.

Speedup math: the full model

The end-to-end speedup of speculative decoding is:

speedup = (1 + E[accepted_per_call]) / (1 + draft_cost_per_call / target_cost_per_call)

Where:

E[accepted_per_call] is the expected number of accepted draft tokens per target verification (between 0 and K).
draft_cost_per_call is the draft's compute cost (much smaller for EAGLE heads than separate draft models).
target_cost_per_call is one target forward pass cost.

For EAGLE-2 with acceptance rate 0.7 and K=5: E[accepted] ≈ 2.5; draft_cost / target_cost ≈ 0.05; speedup ≈ 3.3. For Medusa with E[accepted] ≈ 1.8 and similar draft cost: speedup ≈ 2.6. For vanilla with separate 7B draft on 70B target with E[accepted] ≈ 2.2 and draft_cost ≈ 0.15: speedup ≈ 2.8.

The model explains why EAGLE-3's modest acceptance-rate improvement translates to meaningful end-to-end speedup: the denominator stays small while the numerator grows.

Production case studies (2026)

Anonymised patterns from production deployments:

Frontier lab A: EAGLE-3 head on 100B+ target; throughput improvement around 2.5–3x for chat; routing speculation off for low-batch-size requests.
Frontier lab B: Custom internal speculative decoding (variant of EAGLE) trained jointly with the target.
Top-tier inference provider: vLLM with EAGLE-2 on Llama 70B/405B; throughput improvement around 2x with stable acceptance rates.
Self-host enterprise: vLLM with EAGLE-2 plus FP8 KV cache; near 3x throughput on Llama 70B.
Code-assist provider: speculative decoding with code-trained draft; acceptance rates around 0.8 on code-completion traffic; near 3x throughput.

The pattern: speculative decoding is universal in production frontier inference by mid-2026; the variant choice tracks engineering capacity and target model.

Comparison table: 2026 production defaults

What major serving deployments choose, by workload class.

Workload	Default variant	K range	Notes
Chat (open-domain)	EAGLE-2/3	5–8	Dynamic K helps
Code completion	EAGLE-3 or REST	6–10	High acceptance
Reasoning models	EAGLE-2/3	5–8	Larger gains overall
Structured output (JSON)	Lookahead or constrained EAGLE	3–5	Constraint cost
Vision-language	EAGLE-2 on text path	5–8	Same as chat
Low-latency interactive	Lookahead	3–5	Lowest overhead
Batch / offline	EAGLE-3 + large K	8–16	Throughput-first

Extra FAQ for 2026

Is speculative decoding still worth turning on at very large batch sizes? At very large batch sizes, the target's compute path is more efficient (better tensor-core utilisation), so the relative speedup from speculation shrinks. Modern stacks support adaptive on/off based on batch size. For batch sizes below ~32 on a single GPU, speculation is almost always worth it; above ~128, the answer is workload-dependent.

Does speculative decoding hurt the model's reasoning quality at all? No — the output distribution is provably identical to non-speculative decoding under standard speculative sampling. The model produces the same tokens it would have produced without speculation, just faster.

Why not use a really big draft model (say 30B for a 70B target)? The draft model's cost grows with size. A 30B draft for a 70B target would consume substantial GPU resources without proportional acceptance-rate gains. The sweet spot is typically 1/10 to 1/15 the target size.

Is EAGLE compatible with LoRA-adapted target models? EAGLE heads share the target's hidden state. With LoRA adapters changing the target's behaviour, the EAGLE head's predictions may drift. Practical approaches: retrain the EAGLE head on the LoRA-adapted target, or accept some acceptance-rate degradation.

Can I run speculative decoding on a CPU-only deployment? Yes, llama.cpp supports draft-model speculative decoding on CPU and Apple Silicon. Speedup is real but smaller than on GPU because CPU decode is less HBM-bound.

Does speculative decoding play well with KV cache quantisation? Yes, with caveats. FP8 KV cache + BF16 weights is a common combo. The draft and target both use the same quantised KV. Lower-precision KV (INT4) sometimes has acceptance-rate cost; test in your workload.

How does speculative decoding affect tail latency (P99, P99.9)? Mostly positive — average latency drops, and P99 latency drops too. P99.9 can be slightly worse in some patterns (a request that hits a low-acceptance pocket might run slower than vanilla). For latency-critical workloads, monitor P99.9 explicitly.

Is there a privacy/security implication of speculative decoding? Marginally. The draft model and the target see the same inputs; logs typically capture only the target's outputs (the accepted tokens). The presence of speculation doesn't change the user-visible data flow.

Does speculative decoding help with prefill latency? No. Speculative decoding is a decode-time optimisation. Prefill is already efficient (batched, compute-bound). For prefill latency see chunked prefill and disaggregated inference.

What's the typical engineering effort to enable speculative decoding? For off-the-shelf EAGLE on vLLM: a few hours to evaluate, a few days to qualify in production. For custom drafts or Medusa: weeks of training and validation. For frontier-lab-grade joint training: a multi-month project.

Are there compliance considerations (HIPAA, FedRAMP) with speculative decoding? Speculation is a serving optimisation. It doesn't change compliance posture; the same controls (encryption, audit, residency) apply.

Can speculative decoding help with cost economics? Yes — see AI inference cost economics for the full math. Roughly: 2–3x throughput at near-constant GPU cost equals 2–3x cost reduction per token. The largest providers' inference cost economics are unintelligible without modelling speculative decoding's contribution.

Does speculative decoding help with verifiable inference? The output distribution is identical; verifiability properties (cryptographic attestation, proof systems) carry through speculative decoding cleanly. See verifiable inference for the broader picture.

Will speculative decoding be subsumed by some future technique? Possibly. Continuous batching subsumed iteration-level scheduling; PagedAttention subsumed naive KV management. Speculative decoding may be subsumed by deeper architectural changes (training models that naturally produce multi-token output). For now, the trajectory is more sophisticated speculation, not less.

How does speculative decoding interact with prefix caching? Cleanly. The prefix cache (vLLM v0.6+, SGLang RadixAttention) hits the target's KV. Speculation runs on top of cached prefixes the same way it does on freshly-computed ones. Acceptance rates on cached-prefix requests are similar to non-cached.

Glossary additions for 2026

EAGLE-3: draft head with multi-layer feature aggregation and test-loss training. Successor to EAGLE-2.
MEDUSA-2: target-model self-distillation for multi-token prediction heads.
PASS: pipeline-parallel speculative decoding pattern.
REST: retrieval-based speculative decoding using a datastore of context/continuation pairs.
Lookahead: speculative decoding using only the target via Jacobi iteration.
Dynamic K: adapting the speculation length K based on recent acceptance rate.
Tree decoding: speculation tree with multiple candidate continuations per position.
Acceptance rate: fraction of draft tokens accepted by target verification.
Speedup ceiling: theoretical upper bound on speculative-decoding speedup given workload predictability.

Cross-references

Speculative decoding sits at the centre of the modern LLM serving stack. Related deep dives:

KV cache fundamentals — the memory structure speculative decoding amortises over.
LLM serving in production — the broader serving context.
vLLM, PagedAttention, and continuous batching — the serving substrate speculative decoding runs on top of.
Disaggregated inference — prefill/decode separation and speculative decoding's role.
AI inference cost economics — the cost model that includes speculative decoding.
NCCL guide — communication primitives speculative decoding uses in PP setups.
NVIDIA datacenter GPUs — the hardware tier speculative decoding lives on.
Mixed precision training — related precision concepts for the draft model.
AI training networking — the upstream networking that makes large models trainable.
Verifiable inference — how to attest to speculative-decoded outputs.

Benchmark deep dive: Llama-70B across workload classes

The numbers that matter, by workload class, for a Llama-3.1-70B-Instruct target on H100 80GB with vLLM and EAGLE-2 as the speculative method. Numbers are illustrative and depend on driver, kernel, and exact configuration; treat as "rough order of magnitude" not "exact spec".

Chat (open-domain, temperature 0.7)

Acceptance rate: ~0.65–0.72.
Speedup over vanilla decode: ~2.5–3x.
TTFT impact: negligible (prefill not affected).
ITL improvement: from ~30ms/token to ~12ms/token at batch 16.

Code completion (deterministic, temperature 0)

Acceptance rate: ~0.78–0.85 (code is highly predictable).
Speedup: ~3–4x.
Particularly effective for boilerplate-heavy code and idiomatic patterns.

Math reasoning (Chain-of-Thought)

Acceptance rate: ~0.6–0.7 for reasoning tokens; 0.5–0.6 for final-answer tokens.
Speedup: ~2.2–2.8x.
Stronger speedup if reasoning is long (more tokens = more amortisation).

Creative writing

Acceptance rate: ~0.4–0.55 (high diversity, low predictability).
Speedup: ~1.3–1.8x.
Real benefit, but smaller; some deployments disable speculation for this workload.

Structured output (JSON via constrained decoding)

Acceptance rate: ~0.5–0.65 (constraints reject some drafts).
Speedup: ~1.8–2.4x.
Tuning the draft on constrained examples helps.

Long-context (32k+ input)

Acceptance rate: similar to chat once decode starts.
Speedup on decode portion: similar to chat.
Prefill cost dominates total latency; speculative decoding helps the decode portion.

How EAGLE works inside (architectural detail)

EAGLE's draft is a small transformer head that consumes the target's hidden state and produces token predictions. The architecture:

Feature extractor: a small linear projection of the target's last-layer hidden state.
Draft transformer: typically 1–3 layers, much smaller dimensions than the target.
Output head: produces logits over the vocabulary.
Tree builder: at inference, the draft produces a small tree of candidate continuations.
Verifier: target evaluates the tree in one forward pass; rejection sampling selects accepted paths.

Memory cost: typically 5–10% of target size. Compute cost: typically 1–5% of target compute per call.

The training: EAGLE is trained on rollouts from the target. The draft minimises divergence from the target's distribution at the next-token level. Training data is generated by sampling from the target on a large corpus.

How Medusa works inside

Medusa's heads attach to the target's existing hidden state and predict tokens at offsets +1, +2, +3, ... The architecture:

The target's final hidden state for position N is fed to K=4–8 Medusa heads.
Head k produces a distribution over tokens for position N+k.
Top candidates from each head form a tree of continuations.
Verification: the target evaluates the tree.

Training: in MEDUSA-1, the heads are trained on a corpus while freezing the target. In MEDUSA-2, the target is fine-tuned alongside the heads to improve head calibration.

Memory cost: small (just the heads). Compute cost: per-call modest because heads are tiny.

How Lookahead works inside

Lookahead doesn't add any new model. It exploits the target's ability to evaluate multiple positions in parallel.

The Jacobi iteration:

y_0 = initial guess (e.g., from n-gram cache)
repeat:
  y_{i+1} = target(input, y_i)   # one target call, K positions
until y_{i+1} == y_i              # fixed point

In practice, convergence happens in 2–4 iterations. Each iteration evaluates K positions; speedup is roughly K / (number of iterations).

The n-gram cache: stores recently-seen N-grams from the model's output. On each step, the cache provides initial guesses for the Jacobi iteration. Cache hits dramatically accelerate convergence; misses fall back to neutral initial guesses.

Production checklist for enabling speculative decoding

For teams considering enabling speculative decoding on existing deployments:

Confirm workload class: chat, code, reasoning, structured output, or creative? This determines variant choice and expected speedup.
Pick a variant: EAGLE-2/3 (default), Medusa-2 (if you control training), Lookahead (zero infrastructure).
Acquire or train the draft: HuggingFace has many community-trained EAGLE drafts; some require training.
Stack compatibility check: vLLM, SGLang, TRT-LLM, llama.cpp all support speculation; verify version supports your chosen variant.
A/B test against vanilla: measure end-to-end latency and throughput under representative load.
Verify quality: output distribution should be identical; spot-check on hard prompts.
Tune K: empirical search for optimal K; modern stacks support dynamic K.
Set fallback: above batch threshold, disable speculation to avoid overhead.
Monitor acceptance rate: track production acceptance; if it drops, retrain the draft.
Document: speculation is a non-obvious optimisation; document for operators and on-call.

Quality validation

Run a quality test suite (HumanEval, MMLU subset, internal QA) on both speculation and non-speculation modes.
Token-level identity is provable; quality should be statistically indistinguishable.
Any quality regression indicates a bug or non-standard configuration.

Performance validation

TTFT (Time To First Token): should be unchanged.
ITL (Inter-Token Latency): should drop significantly on decode-bound workloads.
Throughput per GPU: should rise.
P99 latency: should drop or stay flat.

Cost validation

Per-token cost: should drop in proportion to throughput gain.
GPU utilisation: should rise (better tensor-core use during verification).
Memory pressure: should rise slightly (draft KV); check max batch size doesn't shrink dangerously.

Where speculative decoding doesn't help

Workloads where speculative decoding adds little or nothing:

Very small models (1B and below): decode is less HBM-bound; the autoregressive bottleneck is smaller.
Pure prefill workloads: embeddings, classification — no decode.
High-creative, high-temperature workloads: low predictability = low acceptance.
Highly constrained outputs: tight constraints reduce acceptable drafts.
Edge deployments with extreme memory pressure: draft model doesn't fit.

For these cases, focus optimisation efforts elsewhere: chunked prefill, prefix caching, batching, hardware upgrades.

The 2027 outlook

Looking ahead from mid-2026:

EAGLE-3 becomes default in vLLM and SGLang. EAGLE-2 fades.
Tree decoding becomes standard, not optional. Tree width and depth tuned per workload.
Joint draft-target training becomes more common; frontier labs build it into their training pipelines.
Hardware-aware speculation: drafts specifically designed for the hardware's verification path.
Multi-modal speculative decoding: extending to vision and audio outputs.
Architecture co-design: future models trained to be more speculation-friendly natively.
Speculation-aware MoE: experts that align between draft and target.

Speculative decoding for agentic workloads

Agentic workloads (tool use, multi-step planning, code execution) have characteristics that often make speculative decoding particularly effective:

High structural predictability: agent traces follow patterns (think, act, observe; tool call schemas; common follow-up patterns).
Long total token counts: agents generate many tokens per turn, amortising the speedup over more output.
Function-call schemas are highly constrained: drafts trained on the schema accept extremely well.
Tool-use tokens often correspond to known patterns: tool names, parameter formats, JSON outputs are repetitive.

Empirical pattern: speculative decoding speedup on agent workloads is often 2.5–3.5x, higher than open-domain chat. Production agent deployments (Claude Code, Cursor, Devin, OpenAI Operator, Computer Use) all leverage speculative decoding.

Agent-specific tuning

Train the draft on agent traces, not generic web text.
Use higher K when generating tool-call payloads (structured, predictable).
Use lower K when generating natural-language explanations.
Consider tool-specific drafts for high-volume agent traffic.

Interaction with computer-use agents

Computer-use agents generate UI-action tokens that have specific schemas. Drafts trained on these schemas can achieve very high acceptance rates (0.85+) for action generation while remaining at typical rates for free-form reasoning.

Speculative decoding governance and ops

For mature deployments, speculative decoding is an ops surface that requires monitoring and care:

Metrics to track

Acceptance rate (per request, per workload class, per draft version).
Tokens accepted per verification call.
Speedup vs vanilla baseline (canary regression tests).
Verification latency.
Draft inference cost.
Memory headroom (draft + KV).
Failure rate (cases where speculation degraded performance).

Alarms and SLOs

Acceptance rate drops more than 10% from baseline → page.
Speedup drops below threshold → page.
Verification latency rises above SLO → page.
Memory pressure threshold approached → automatic disable.

Draft lifecycle

Drafts have a lifecycle (training, validation, staging, production).
Like other model artifacts, they should be versioned and rollback-able.
When the target model is updated, the draft typically must be updated too.
Documenting the draft training pipeline is essential for operability.

Cost attribution

Speculative decoding's cost savings should be attributed properly in capacity planning.
A team enabling speculation should expect ~50% reduction in serving fleet for the same workload, give or take.

Open-source draft model ecosystem

The open-source ecosystem for speculative-decoding drafts has matured. Key sources:

HuggingFace community drafts: many EAGLE and Medusa drafts published by community researchers; quality varies.
vLLM / SGLang model zoo: vetted drafts that ship with the stack.
Llama-3 family: official EAGLE drafts exist for Llama-3.1 8B/70B/405B.
Qwen / DeepSeek / Mistral: community drafts; coverage varies.
Custom drafts: many large deployments train their own.

Practical advice: start with a vetted draft (vLLM's recommended), benchmark on your workload, decide whether to train custom.

Training a custom draft (rough cost)

For an EAGLE draft on a 70B target:

Training data: ~1B–10B tokens of representative output.
Training compute: ~1–10k H100-hours.
Engineering time: 2–8 weeks depending on team experience.
Validation: another 1–4 weeks.

For teams without this budget, off-the-shelf drafts are the pragmatic choice.

Composing optimisations

In production, speculative decoding composes with many other optimisations:

Continuous batching: speculation runs inside a continuously batched scheduler.
PagedAttention: speculation uses paged KV; draft uses its own pages.
Prefix caching: speculation runs on cached prefixes.
Chunked prefill: speculation engages when decode begins.
FP8 weights / KV: speculation works on quantised models.
LoRA / multi-LoRA: speculation needs draft compatibility; sometimes per-LoRA drafts.
Disaggregated serving: speculation lives in decode pool.
Structured outputs: speculation respects constraints.
Tool use: speculation accelerates tool-call generation.

The compositions interact non-trivially. The largest gains usually come from enabling all of these together; modern stacks (vLLM v1, SGLang) make this composition practical.

For the bigger picture see LLM serving in production and vLLM and PagedAttention.

Final synthesis on speculative decoding

The state of speculative decoding in mid-2026:

EAGLE-2 is the production default; EAGLE-3 rolling out.
Lookahead and Medusa-2 are second-tier choices for specific configurations.
Stack support is mature across vLLM, SGLang, TRT-LLM.
Speedups of 2–3x are typical; higher for code and reasoning workloads.
The technique composes cleanly with the rest of the serving stack.

The trajectory through 2027:

EAGLE-3 becomes default.
Tree decoding standardises.
Joint draft-target training at frontier labs.
Acceptance rate models become more workload-aware.
Speculation-aware MoE architectures may emerge.

For practitioners:

Enable speculative decoding on decode-bound workloads.
Monitor acceptance rate and tune K.
Use mature stacks; don't roll your own.
Compose with prefix caching, continuous batching, paged KV, FP8.

The end result: throughput improvements that materially change inference cost economics, with no quality compromise. For most production deployments, speculative decoding is no longer an optional optimisation; it's part of the baseline.

Practical guide: enabling EAGLE-2 on vLLM

A step-by-step practical guide for the most common deployment scenario.

Prerequisites

vLLM v0.6.x or later (v1 preferred).
Target model checkpoint (Llama 3.1 70B, etc.).
Pre-trained EAGLE draft for the target.
H100 / H200 GPUs (FP8 path benefits significantly).

Configuration

CLI arguments:

vllm serve <target-model> \
  --speculative-model <eagle-draft> \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1 \
  --use-v2-block-manager

Validation

Run a quality test on the baseline (no speculation).
Run the same test with speculation enabled.
Verify outputs are statistically indistinguishable.

Performance measurement

Run benchmark suite (sharegpt-like prompts) on baseline.
Run on speculation.
Compare TTFT, ITL, throughput.
Expect 2–3x throughput improvement on chat workloads.

Tuning

Try K=3, 5, 8; find the sweet spot for your workload.
Monitor acceptance rate.
Adjust based on workload characteristics.

Operationalisation

Set up monitoring for acceptance rate.
Configure fallback (disable speculation above threshold batch size).
Document configuration for ops.

Common pitfalls

Mismatched draft and target (different vocab; check carefully).
Quality regression due to bug in draft (validate quality before deploying).
Memory pressure (draft KV adds; monitor).
Configuration version skew between vLLM versions.

Table of contents

Key takeaways

Mental model: speculative decoding in one minute

The decoding landscape in 2026

Where each technique fits

Why decode is slow

The core algorithm

Acceptance rate

Variants: vanilla, EAGLE, MEDUSA, Lookahead, REST

Vanilla speculative decoding (Leviathan 2023, Chen 2023)

EAGLE / EAGLE-2 (Li 2024)

MEDUSA (Cai 2024)

Lookahead decoding (Fu 2024)

REST (Retrieval-based)

Comparison

Pick by workload

EAGLE-3 and the 2026 frontier

Side-by-side acceptance benchmarks (Llama-3.3-70B, May 2026)

Draft length K: tuning

The tradeoff

How acceptance probability decays with K

Auto-tuning

Per-request K vs global K

K vs tree width tradeoff (EAGLE-2/3)

Acceptance rate by sequence position

KV cache implications

Target model KV

Draft model KV (vanilla and EAGLE)

Combined with prefix caching

Combined with FP8 KV

When spec-decode wins

Highly predictable workloads (2–3× speedup)

Moderately predictable (1.5–2× speedup)

Low-predictability (<1.5×, sometimes <1×)

Rule of thumb

When it doesn't

Memory-constrained deployments

Very small target models (under 30B)

Production multi-tenant with diverse workloads

Speculative loops in agents

Stack support and configuration

vLLM

SGLang

TensorRT-LLM

TGI

LMDeploy

Stack maturity matrix

Worked examples

Example 1: code-completion service

Example 2: customer support chat

Example 3: creative writing

Example 4: agent / tool use

Recent research directions

Tree-based drafts

Cross-request speculation

Adaptive K within a single request

Speculative decoding for reasoning models

Quantized drafts

A short history of speculative decoding

Mathematical correctness

Rejection sampling derivation

Why both probabilities are needed

Temperature interaction

Quality preservation guarantee

Architecture deep dive: EAGLE-2

Draft model architecture

Tree-structured drafting

Dynamic tree shape

Verification batching

Acceptance rate empirics

Architecture deep dive: MEDUSA

Multiple heads

Verification

Trade-offs vs EAGLE

Lookahead decoding

Mechanism

Speedup

When to use

REST: Retrieval-based speculative decoding

Mechanism