LLM Serving: The Complete Guide

Q: Should I use vLLM or SGLang?

Default vLLM. Switch to SGLang specifically when you have heavy prefix sharing, need their structured-output features, or do agentic workloads.

Q: When does TensorRT-LLM make sense?

NVIDIA-only, one primary model, deploying at scale, specifically need maximum throughput.

Q: How do I handle a 1M-token context?

Hybrid architectures (Jamba), MLA-based models (DeepSeek), aggressive sparse attention, hierarchical KV with NVMe offload, or simply not. Use the architecture, not the optimization.

Q: Should I use disaggregated prefill/decode?

If you have very skewed prefill:decode ratios (RAG with 32k input, 200 output), maybe. ~30% cost saving, operational complexity is the trade.

Q: Why does my P99 TTFT spike randomly?

Long-context requests blocking batch, eviction events, Python GC pauses, NCCL hiccups. Investigate one at a time.

Q: Can I run a quantized model in production?

Yes. FP8 weights + FP8 KV is the modern default on Hopper.

Q: How do I scale serving for a sudden 10× traffic spike?

You don't, fully. GPU cold start is 60-180s. Pre-warmed pool, backpressure, graceful degradation.

Q: How do I serve different fine-tunes efficiently?

Multi-LoRA. One base model, dynamic adapters.

Q: What about CPU inference?

7B-class feasible (10-30 tok/sec on beefy CPU). 70B+ painfully slow. llama.cpp is canonical.

Q: How do I migrate stacks?

In phases. Stand up new stack in parallel, replay recorded trace, compare, ramp.

LLM serving is its own discipline now. The mechanisms — continuous batching, paged KV, prefix caching, speculative decoding, multi-LoRA, scheduling — are well-defined enough to reason about precisely instead of treating "the inference server" as a black box. Most application engineers still don't, and they pay for it. This reference shows how to pick the right stack, size capacity, debug latency tails, and figure out which optimization actually pays off in your case. Serving is the inference half of the stack — if that framing is new, start with training vs inference and what a context window is, since KV-cache sizing hinges on it.

Key takeaways
Mental model: LLM serving in one minute
What "serving" actually means
The two phases: prefill and decode
Continuous batching: the headline win
PagedAttention and the KV cache layer
Prefix caching and RadixAttention
Speculative decoding
Quantization at serving time
Multi-LoRA serving
Scheduling, admission control, and priority
Multi-GPU: TP, PP, EP, DP combinations
The major stacks compared
Latency engineering: prefill, decode, tails
Capacity planning
Cost economics
Autoscaling and traffic shaping
Observability and SLO design
Streaming, tool use, structured output
Failure modes and incident response
Serving stack feature matrix
Latency budget breakdown
SLO design and queueing math
Production debugging playbook
The bottom line
FAQ
Extended FAQ
Glossary
References
PagedAttention mechanics deep dive
Continuous batching scheduler in detail
Prefix caching mechanics
FlashAttention-3 paged kernel
Per-feature matrix (deep)
KV quantisation in serving
MoE serving in detail
Vision-language model serving
Throughput vs latency math
SLO design across percentiles
Failure mode taxonomy
Observability deep dive
Deployment patterns deep dive
Cost arithmetic per stack
Benchmarks per stack
When to roll your own
Future direction
Cross-references
Extra FAQ for serving in 2026
Production case studies (2026)
Disaggregated prefill/decode in production
Long-context serving
Reasoning model serving
Multi-model serving
Streaming patterns
Structured output and tool use serving
Safety and guardrail integration

Key takeaways

LLM serving is the discipline of converting a model file and a stream of incoming requests into output tokens efficiently. The mechanisms that matter:

Continuous batching dynamically merges new requests into the in-flight batch as decode progresses, an idea introduced by Orca (Yu et al., OSDI 2022). 2–4× throughput vs static batching.
PagedAttention divides the KV cache into fixed-size blocks, eliminating fragmentation (Kwon et al., arXiv:2309.06180). Lifts effective KV utilization from 30–50% (naive) to 90%+ (paged).
Prefix caching dedupes KV blocks across requests sharing prompt prefixes. 2–10× throughput on chat with shared system prompts.
Speculative decoding generates K candidate tokens with a draft model and verifies in one target-model pass. 2–3× decode throughput on agentic workloads.
Multi-LoRA serving runs many fine-tuned adapters on one base model concurrently. Eliminates the "one replica per fine-tune" memory bloat.

The major stacks in 2026: vLLM (default safe choice), SGLang (best for chat with shared prompts), TensorRT-LLM (max throughput on NVIDIA), TGI (HF ecosystem), LMDeploy (Chinese open-weight specialist), llama.cpp (local/CPU/edge). For reasoning-model serving, MoE serving, and agent serving infrastructure, the same primitives apply with different scheduling profiles.

The non-obvious thing: serving optimization compounds. You don't pick one of the techniques above; you stack them. The 4–24× headline numbers from the original vLLM paper come from the product of paging × continuous batching × prefix caching × good scheduling, not any single mechanism.

Mental model: LLM serving in one minute

Two named problems run the entire field. The first is head-of-line blocking: static batching pads every request in a batch to the longest one, so a 1k-token reply forces 31 other 30-token replies to wait — the GPU sits half-idle and tail latency explodes. The second is KV fragmentation: each request reserves a contiguous slab of HBM for its KV cache sized for the worst-case sequence length, but most requests use a fraction of that, so 50-70% of expensive HBM is reserved-and-empty. Together they cap GPU utilization at roughly 30%, which is why a vanilla HuggingFace generate() loop on an H100 costs 5-10x more per million tokens than vLLM.

The fix is two ideas borrowed wholesale from operating systems. Continuous batching is preemptive scheduling: each decode step the scheduler re-forms the batch, evicting finished sequences and admitting new ones at the same tick — no request waits for the slowest sibling. PagedAttention is virtual memory: the KV cache is sliced into fixed 16-token blocks, requests hold a logical-to-physical block table, and HBM is allocated lazily, page by page. Fragmentation collapses, sharing becomes cheap, and prefix caching falls out for free because two requests with the same prompt point at the same physical blocks.

Aspect	Static batching, contiguous KV	Continuous batching + PagedAttention
Batch composition	Fixed at admission	Re-formed every decode step
KV allocation	Contiguous, worst-case sized	16-token blocks, lazy
KV utilization	30-50%	90%+
Tail latency	Bounded by longest reply	Bounded by per-step cost
Throughput vs naive	1x	2-24x (vLLM paper range)
When it pays off	Almost never	Always above batch size 2

Conceptually:

# Naive: pad to longest, run K decode steps, release
batch = [pad(x, max_len) for x in requests]
for _ in range(max_len): batch = step(batch)

# vLLM: one call, scheduler + paged KV handle everything
engine = LLM(model="meta-llama/Llama-3-70B")
engine.generate(prompts, sampling_params)

One number to remember: PagedAttention lifts effective KV cache utilization from 30-50% (naive contiguous) to 90%+, and stacked with continuous batching delivers 2-24x throughput on real serving traces (Kwon et al., SOSP 2023).

The rest of this guide is everything that extends or depends on that idea — prefix caching, speculative decoding, multi-LoRA, scheduling, and the stack-by-stack comparison.

What "serving" actually means

A serving stack does five things:

Receives requests over HTTP, gRPC, or some socket protocol. Most expose an OpenAI-compatible REST API by 2026.
Tokenizes inputs — converts text/images into the integer token IDs the model consumes. Often a non-trivial cost (BPE for long prompts can take 10ms+).
Schedules requests onto GPU resources — decides which requests to batch, which to evict, which to preempt.
Runs the model forward pass — prefill and decode, layer by layer, with whatever optimizations the stack provides (paging, FlashAttention, fused kernels).
Streams tokens back — usually as Server-Sent Events for OpenAI-compatible APIs, with proper chunked transfer encoding.

A serving stack is not:

The model itself. It's the runtime around the model.
A fine-tuning pipeline. Different concern, different tools.
A retrieval system. Some stacks have plugin points for retrieval, but RAG is its own discipline.
A model registry. You point a serving stack at a checkpoint; the registry is upstream.

The line between "the serving stack" and "the inference engine" gets blurry. vLLM and SGLang are typically used as both: HTTP server + inference engine in one process. TensorRT-LLM is more often used as just the engine, with Triton Inference Server providing the HTTP layer. Hugging Face TGI bundles both. The right granularity for your team depends on operational preferences.

What follows assumes you're operating an end-to-end stack: incoming requests → output tokens.

The two phases: prefill and decode

A single LLM request has two distinct compute phases with very different cost profiles:

Prefill: process the entire input prompt to populate the KV cache and produce the first output token. Compute scales as O(N²) for naive attention, O(N) for memory bandwidth-bound dense ops. Heavily compute-bound: GPU utilization is high.

Decode: generate output tokens one at a time, each step reading the full KV cache and producing a new token. Compute per step is fixed (one new token, regardless of context); memory bandwidth dominates because you're re-reading the entire KV cache every step. Heavily memory-bound: GPU utilization is low without batching.

Concrete numbers on Llama-3 70B FP8 on H100:

Phase	Compute pattern	TFLOPs/sec achieved	HBM bandwidth used
Prefill, 4k tokens	Compute-bound matmul	~600 (75% of peak)	~1 TB/s (33% of peak)
Decode, single token	Memory-bound	~30 (4% of peak)	~3 TB/s (95% of peak)

This asymmetry is the structural reason continuous batching helps. In prefill you're already compute-saturated; batching does little. In decode you're memory-saturated and using ~5% of compute; batching many requests together is essentially free additional compute.

Prefill cost grows quadratically with prompt length. Decode cost grows linearly with cache size. For a 32k-token prompt + 1k-token output:

Prefill: ~500ms on Llama-3 70B FP8 on 2× H100 with FlashAttention (Dao et al., arXiv:2205.14135; FlashAttention-2, arXiv:2307.08691).
Decode: 1000 tokens × 30ms/token = 30s for the output.

Time to first token is dominated by prefill. Tokens per second after that is dominated by decode efficiency. These are different optimization problems.

Chunked prefill

A practical complication: a single 64k-token prefill takes 1+ seconds and blocks every other request waiting in the batch. Modern stacks split prefill into chunks (e.g., 2k tokens at a time) and interleave them with decode steps. This levels out latency at the cost of slight prefill efficiency loss.

vLLM enables chunked prefill with --enable-chunked-prefill (default in v0.6+). SGLang has similar. The setting matters for tail latency: with it, P95 first-token latency stays bounded even when long-context requests are in flight.

Disaggregated prefill and decode

A 2025 development: separate the prefill and decode phases onto different hardware — see our deep dive on disaggregated inference. Prefill servers handle the compute-heavy work; decode servers handle the memory-bandwidth-heavy work. The two communicate by transferring the KV cache between them. Foundational systems work here includes DistServe (Zhong et al., arXiv:2401.09670), Splitwise (Patel et al., arXiv:2311.18677), and Mooncake (Qin et al., arXiv:2407.00079). Speculative decoding (Leviathan et al., arXiv:2211.17192) — covered in our speculative decoding post — and shared-prefix techniques like SGLang's RadixAttention (Zheng et al., arXiv:2312.07104) stack on top.

The pitch: each phase runs on hardware sized for its actual bottleneck. Prefill on compute-dense GPUs, decode on memory-dense GPUs.

How the disaggregation works mechanically

Client → API gateway
            ↓
        Prefill cluster (compute-optimized, e.g., H100 FP8)
            ↓
        KV cache transfer (RDMA / NVLink / shared memory)
            ↓
        Decode cluster (memory-bandwidth-optimized, e.g., H200)
            ↓
        Stream tokens back to client

Critical: the KV cache transfer between prefill and decode clusters has to be fast. For Llama-3 70B at 32k context, that's ~10 GB to transfer per request. Over 400 Gb/s InfiniBand: 200ms — non-trivial.

Optimizations:

Co-located clusters: prefill and decode hardware in the same row, on the same NVLink/InfiniBand fabric. Transfer time drops to tens of milliseconds.
Layered KV transfer: stream KV by layer as prefill produces it. Decode can start before all KV is transferred.
Memory-mapped shared KV: prefill and decode share memory regions. Eliminates the transfer entirely on co-located setups.

When disaggregation pays off

The win comes from running each phase on cost-optimized hardware:

Prefill: 100% compute-bound. Wants maximum FLOPS per dollar. H100 FP8 is sweet spot.
Decode: 95% memory-bandwidth bound. Wants maximum HBM bandwidth per dollar. H200 sweet spot.

For a workload with 50/50 prefill/decode time split:

Co-located on H100 only: utilization ~70% (one phase always under-utilized).
Co-located on H200 only: pays H200 premium for prefill it doesn't need.
Disaggregated H100 prefill + H200 decode: each phase ~95% utilized. ~30% cost reduction.

The win is larger for workloads with skewed prefill:decode ratios:

Long-context RAG (32k input, 200 output): prefill dominates. Disaggregation makes sense.
Reasoning models (4k input, 8k output): decode dominates. Less benefit; co-located decode-optimized hardware is fine.
Chat (1k input, 200 output): relatively balanced. Disaggregation helps but less.

Stack support in 2026

NVIDIA NIM: official disaggregation extension. Production-grade.
Microsoft research: published disaggregation papers; some implementations open-source.
vLLM forks: experimental. Not yet mainline.
Open-source production stacks: not yet standard. Most teams still co-locate.

For most production deployments in 2026, co-located is fine. Disaggregation is a 30% cost optimization that requires meaningful infrastructure investment — worth it at >100M tokens/month, marginal below.

Continuous batching: the headline win

Continuous batching (Yu et al., Orca, 2022) is the single most important serving idea of the LLM era. Without it, vLLM and SGLang's other optimizations would matter much less.

Static batching: the baseline

The naive approach: collect N requests, batch them, run forward, return outputs. Repeat. Static batching has two big problems:

Tail-blocked. If 8 requests are batched and one wants 4000 output tokens while the other 7 want 100 each, the first 7 finish in a few seconds and the batch hangs around for the 8th. Six GPUs of capacity wasted on idle waiting.

Bursty. New requests arriving during a batch can't join. They wait in the queue until the current batch completes. Latency under bursty load is terrible.

Continuous batching: the fix

Instead of running batches to completion, the scheduler operates one decode step at a time, dynamically deciding which sequences to include in each step:

Each step: gather all currently-active sequences (those still generating).
Run one forward pass producing one new token for each.
After the step: remove finished sequences, accept new ones from the queue.
Repeat.

The result: as soon as one request finishes its 100 tokens, the freed slot is given to a new incoming request. The 4000-token request keeps decoding alongside fresh arrivals. Throughput stays high regardless of output-length variance.

How much it actually buys

From the original Orca paper and many reproductions:

Static batching, mixed output lengths (50–4000 tokens): GPU utilization 30–50%.
Continuous batching, same workload: GPU utilization 85–95%.

That's a 2–3× throughput improvement on its own, before any KV optimizations.

Continuous batching is now the default in vLLM, SGLang, TensorRT-LLM, TGI, and LMDeploy. If you find a stack in 2026 that uses static batching, it's a legacy artifact.

Configuring continuous batching

Most stacks expose two main knobs:

max_num_seqs (or max_batch_size): hard cap on concurrent sequences. Larger = higher throughput, more memory pressure, longer scheduler overhead.
max_num_batched_tokens: cap on total tokens processed per step (decode + chunked prefill). Larger = better GPU utilization, longer per-step latency.

Sane defaults for production:

vLLM Llama-3 70B on 2× H100: max_num_seqs=64, max_num_batched_tokens=8192.
For chat-heavy workloads: bump max_num_seqs higher (128, 256).
For long-context-heavy workloads: keep max_num_seqs lower (16-32) and bump max_num_batched_tokens.

Auto-tuning is on the way (vLLM 0.7+ has experimental adaptive scheduling). For now, manual tuning based on workload measurement.

How continuous batching interacts with prefill

A subtle issue: prefill is compute-heavy and one prefill step can take hundreds of milliseconds for long contexts. If you naively interleave prefills with single-token decode steps, the decode users wait while prefill blocks the GPU.

Stacks handle this in three patterns:

Prefill-first scheduling: when a new request arrives, run its prefill before continuing decode. Simple, but causes latency spikes for in-flight users when many new requests arrive simultaneously.

Chunked prefill (vLLM 0.6+, SGLang): split a request's prefill into chunks (e.g., 2k tokens per chunk) and interleave with decode. Each iteration mixes some chunked prefill work with decode work for in-flight users. Smoother latency, slight prefill efficiency loss.

Disaggregated prefill (NVIDIA NIM, research stacks): prefill and decode run on different GPUs. The KV cache transfers between them. Each phase runs on hardware sized for its bottleneck.

For most production workloads, chunked prefill is the right default. Pure prefill-first is acceptable for low-concurrency single-tenant serving. Disaggregation pays off only at very large scale.

Scheduler internals

The scheduler runs once per iteration. Its responsibilities:

def schedule_iteration():
    # 1. Free blocks for finished sequences
    for seq in finished_sequences:
        free_kv_blocks(seq)
    
    # 2. Decide which queued requests to admit
    while queue and can_admit(queue.peek()):
        seq = queue.pop()
        allocate_kv_blocks_for_prefill(seq)
        active_sequences.add(seq)
    
    # 3. Decide which active sequences need preemption
    while not enough_kv_for_decode_step(active_sequences):
        victim = pick_preemption_victim(active_sequences)
        preempt(victim)  # evict KV, return to queue
    
    # 4. Run one forward pass for active sequences
    forward_pass(active_sequences)
    
    # 5. Update sequence states; mark finished
    for seq in active_sequences:
        if seq.last_token == EOS or seq.length >= max_length:
            mark_finished(seq)

Critical decisions in pick_preemption_victim:

vLLM's default: longest-running sequence (give priority to short responses).
vLLM's priority extension: highest-priority sequences are protected from preemption.
SGLang's RadixAttention: tries to preempt sequences that don't share prefixes with active ones (preserves cache value).

The pre-emption policy can dramatically affect tail latency under load. The default rarely needs tuning; if you're seeing high preemption rates and tail latency, consider lowering max_num_seqs instead.

Continuous batching's memory management

The challenge with continuous batching: every iteration potentially has a different active set. Memory has to be efficient regardless.

Three coordination mechanisms:

Pre-allocation: the scheduler pre-allocates KV blocks during admission. If a request can't fit, it waits in the queue. Avoids mid-step OOM but wastes memory if the request finishes early.

Lazy allocation: blocks are allocated on demand as the sequence grows. Better memory efficiency but requires the allocator to be safe under concurrent steps.

Hybrid: pre-allocate enough for prefill + some decode buffer; lazily allocate beyond. Most modern stacks use this.

vLLM's PagedAttention enables hybrid because the block-based allocator handles fragmentation cheaply.

PagedAttention and the KV cache layer

PagedAttention is covered in depth in the KV Cache guide. The brief version:

The KV cache stores per-token key and value vectors for every layer of attention. Naive contiguous allocation produces 30–50% utilization due to internal and external fragmentation. PagedAttention divides KV into fixed-size blocks (typically 16 tokens) and maintains per-sequence block tables — exactly the OS virtual-memory pattern. Utilization jumps to 90%+.

For serving specifically, the things to know:

Block size is a tunable. block_size=8 minimizes tail waste (~8 tokens/sequence wasted), block_size=16 is the default sweet spot, block_size=32 or larger helps long-context-heavy workloads with marginally better kernel locality.

Eviction policy matters at saturation. When the KV is full, your stack has to evict. vLLM uses recompute-based preemption (evict, restart prefill on reschedule). SGLang's RadixAttention often dodges this by sharing aggressively. TRT-LLM supports swap-based preemption (evict to host memory, swap back). Pick based on your headroom.

Quantize the KV. FP8 KV halves memory at near-zero quality cost on most workloads. INT4 KV is workload-dependent (breaks long-context retrieval). Enable on vLLM with --kv-cache-dtype fp8_e4m3 --calculate-kv-scales.

The compounding effect: paged + quantized KV gives you 4× the in-flight requests at the same memory budget vs naive contiguous BF16 KV.

KV cache lifecycle in a serving stack

A request's KV cache goes through specific states:

[REQUEST ARRIVES]
       ↓
   [QUEUED] — waiting for KV blocks
       ↓
   [PREFILL] — KV being computed for the prompt
       ↓
   [DECODE] — KV growing one block per N tokens
       ↓
   [FINISHED] — blocks freed (or kept for prefix caching)

State transitions:

Queued → Prefill: when scheduler admits the request and allocates initial blocks.
Prefill → Decode: after the prompt is fully processed.
Any → Preempted: when KV is full and the request is the eviction victim.
Preempted → Queued: re-enters the queue for re-admission.
Decode → Finished: when EOS is generated or max_tokens reached.

For prefix caching, finished blocks may be retained in a "free blocks with hash" pool: they're available to be reclaimed by new requests, but if a request arrives with the same prefix, the blocks are reused.

Block table data structure

Each sequence has a block table — an array mapping logical positions to physical block IDs:

class BlockTable:
    sequence_id: int
    blocks: list[int]  # physical block IDs in logical order
    
    def get_kv_address(self, token_position):
        block_idx = token_position // BLOCK_SIZE
        offset = token_position % BLOCK_SIZE
        physical_block = self.blocks[block_idx]
        return BLOCK_BASE_ADDRESS + physical_block * BLOCK_BYTES + offset * KV_PER_TOKEN_BYTES

The block table is small — for 32k tokens with block_size=16, that's 2000 entries × 4 bytes = 8 KB. Fits in GPU L2 cache.

Attention kernels read the block table for every attention computation. Modern paged-attention kernels (FlashAttention-3) load the entire block table once into shared memory at the start of attention, avoiding repeated indirection per layer.

Prefix caching and RadixAttention

Block-level prefix sharing: when two requests share prompt prefix tokens, they share the underlying KV blocks instead of computing them twice. Free throughput on workloads with prefix overlap.

Where prefix caching pays

Chat with shared system prompts: 100 users sharing a 1k-token system prompt = 95% prefix hit rate, 4–6× effective KV capacity.
Multi-turn conversations: each new turn shares prior turns. Essentially 100% hit rate per session.
Few-shot prompting with stable examples: 85–95% hit rate.
Code completion within an editor session: 80–90% hit rate.

Where it doesn't

RAG with diverse retrieval: prefix is mostly the retrieved context, which varies per query. Hit rate 10–30%.
Single-turn queries with no system prompt: hit rate ~0%.
Random sampling at API layer that adds entropy to prompts: kills the hit rate.

vLLM's block-level vs SGLang's RadixAttention

vLLM's --enable-prefix-caching (default v0.6+) implements block-level prefix sharing: when a new request arrives, its prefix blocks are checked against a hash table; matching blocks are reused.

SGLang's RadixAttention generalizes this to a full radix tree. Every distinct prefix is one node; every divergence is a new branch. New requests do longest-prefix-match against the tree, mount at the matched node, only compute the suffix. Eviction is LRU at the leaf level; shared internal nodes are protected.

The functional difference: SGLang's tree-based approach handles deep prefix hierarchies more efficiently than vLLM's hash-based approach. For most workloads the two are within 10% of each other; for chat with deep multi-turn conversation history, RadixAttention can be 2–3× better.

Things that break prefix caching invisibly

Adding a timestamp to your system prompt ("It is currently May 7, 2026 at 3:42 PM"). Every request has a different prefix.
Embedding session IDs in prompts.
Tokenizer changes mid-deployment (cached blocks reference stale token IDs).
Random temperature sampling injected at the prompt level.

Cross-replica prefix caching

In multi-replica deployments, prefix cache is per-replica unless you load-balance with consistent hashing. Round-robin load balancing with N replicas approximately divides hit rate by N. If you have stable prefix patterns and want to maximize sharing, route by prefix hash instead of round-robin.

Speculative decoding

Speculative decoding generates K candidate tokens with a small "draft" model, verifies them in one pass through the large "target" model. If accepted, you got K tokens for ~1 target-pass + K cheap draft-passes. If rejected, fall back to standard decode.

How it works

Maintain target model (e.g., Llama-3 70B) and draft (e.g., Llama-3 8B or a model-specific small head).
At each step:
- Draft generates K candidate tokens.
- Target processes input + K candidates in one forward pass.
- Accept the longest prefix of candidates whose probabilities match the target's distribution.
Output the accepted prefix; redo if all rejected.

The math: spec-decode preserves the target distribution exactly. There's no quality loss.

Variants

EAGLE-2 (Li et al., 2024): the dominant variant in 2026. The draft is a small "head" that shares the target's hidden states. Compute is essentially free; memory adds ~10–15% to KV.

MEDUSA (Cai et al., 2024): adds multiple decoding heads to the target model itself. No separate draft. KV is unchanged. Less aggressive speedup but no extra memory.

Lookahead decoding (Fu et al., 2024): the target model itself drafts via lookahead steps. Modest speedup.

When spec-decode wins

Agentic workloads: 2–3× speedup. Output is highly predictable.
Code completion: 2.5–3× speedup. Code is repetitive enough.
Chat with consistent style: 1.5–2× speedup.

When it doesn't

Creative writing with high entropy: 1.0–1.3×. Draft accuracy poor.
Small models (under ~30B): draft and target too close in capability.
Highly KV-constrained deployments: extra KV may force smaller in-flight batch.

Configuring spec-decode (vLLM)

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --use-v2-block-manager

K (num-speculative-tokens) is the main knob:

K=2–3: low risk, low reward.
K=4–5: typical sweet spot.
K=6–8: high reward when draft accuracy is high.

vLLM 0.7+ auto-tunes K based on observed acceptance rates.

Quantization at serving time

Two distinct quantizations matter for serving:

Weight quantization reduces model memory:

FP8 (e4m3): ~0.1 point quality cost. Halves weight memory. Default for new deployments on Hopper/Blackwell.
INT8 (W8A16): similar quality, similar memory. More mature on older hardware.
AWQ INT4: ~0.5 point quality cost; quarters weight memory. Sweet spot for memory-bound deployments.
GPTQ INT4: similar to AWQ.
NF4 / Q4_K_M (llama.cpp): INT4 variants tuned for CPU/edge.
FP4 (Blackwell): emerging. 2× tensor core throughput vs FP8. Quality data still preliminary.

KV cache quantization reduces per-token cache memory. Decided independently. See the KV cache guide.

Stack support matrix

Stack	FP8 W	INT8 W	AWQ INT4	GPTQ INT4	FP4 W (Blackwell)
vLLM 0.6+	✅	✅	✅	✅	⚠ early
SGLang	✅	✅	✅	✅	⚠ early
TRT-LLM	✅	✅	✅	✅	✅
TGI	✅	✅	✅	✅	❌
LMDeploy	✅	✅	✅	✅	⚠ early
llama.cpp	partial	✅	❌	partial	❌

Choosing a format

Memory-bound? Use INT4 (AWQ).
Memory comfortable? Use FP8.
On Blackwell with FP4 support? FP4 if quality cost is acceptable.
Latency-sensitive on Hopper? FP8 KV + FP8 weights.
Older hardware (Ampere)? INT8 weights, BF16 KV.

Multi-LoRA serving

LoRA (Hu et al., 2021) trains low-rank matrix updates added to specific weight matrices. The base model is frozen; the LoRA adapter is small (~1% of full-model size, often hundreds of MB).

For serving, LoRA matters because: many fine-tunes can share one base model. You don't need a separate replica per fine-tune.

How it works

In a multi-LoRA setup:

The base model is loaded once into GPU memory.
LoRA adapters are loaded into a small adapter pool.
Each request specifies which LoRA to use.
Inference computes base-model forward + LoRA-specific adjustment.

The LoRA-specific adjustment is computed efficiently: instead of materializing the full adjusted weight matrix, the LoRA decomposition W' = W + AB is applied at compute time, where A and B are small.

Stack support

Stack	Multi-LoRA	Notes
vLLM	✅	Production-grade, dynamic adapter loading
SGLang	✅	Solid
TRT-LLM	✅	NVIDIA-internal
TGI	✅	Mature
LMDeploy	⚠ partial	Improving
llama.cpp	⚠ via merge	Less convenient

Performance and operational implications

LoRA has a small but non-zero compute cost: each layer with a LoRA adapter does an extra small matmul. Throughput cost: 5–15% vs base alone, depending on layer count.

Memory cost per adapter: 50–500 MB depending on rank. A few hundred LoRAs fit in modest GPU memory.

The big win is operational: instead of running 50 replicas (one per fine-tune), you run a few replicas of the base model and dynamically load LoRAs per request. ~10× cost reduction for multi-tenant fine-tune-heavy workloads.

Configuring on vLLM

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --enable-lora \
  --max-loras 8 \
  --max-lora-rank 64 \
  --lora-modules adapter1=path/to/adapter1 adapter2=path/to/adapter2

Per-request:

{"model": "adapter1", "messages": [...]}

When multi-LoRA isn't right

LoRAs that target many layers reduce throughput advantage.
Few fine-tunes (1–2) on heavy traffic — dedicated replicas may be simpler.
"Fine-tunes" that actually need different base models — multi-LoRA doesn't help.

For most enterprise multi-tenant scenarios with one base + many adapters, multi-LoRA is straightforwardly better.

How multi-LoRA actually works under the hood

The two main implementations are S-LoRA (Sheng et al., 2024) and Punica (Chen et al., 2024). Both achieve similar goals via slightly different mechanisms.

S-LoRA's approach: maintains a unified KV cache for all LoRA-adapted requests, and applies LoRA computation as an extra pass after the base model's attention/MLP. The trick is fusing many small LoRA computations into batched operations.

Punica's approach: introduces SGMV (Segmented Gather Matrix-Vector multiplication) — a custom CUDA kernel that handles requests with different LoRA adapters in a single batched operation. Each request's LoRA weights are gathered from a unified pool just before their multiplication.

Both libraries handle the core challenge: you have many requests in a batch, each potentially using a different LoRA adapter. The base model's matmul is shared; the LoRA adjustments differ per request.

The compute pattern (batched, simplified):

# Base forward (all requests, shared base weights)
hidden = base_layer(input)

# LoRA adjustment (per-request adapters)
for lora_id, lora_indices in active_loras:
    hidden[lora_indices] += lora_a[lora_id] @ lora_b[lora_id] @ input[lora_indices]

S-LoRA / Punica fuse this loop into a single GPU kernel for efficiency.

LoRA rank tradeoffs

A LoRA adapter has rank r. Higher rank = more expressive adapter, more memory, more compute per request.

r=8: minimal capacity, ~50 MB per adapter. Used for narrow specialization.
r=16-32: standard. ~100-200 MB per adapter.
r=64-128: high capacity, ~400-800 MB. Closer to full fine-tune in expressivity.
r=256+: rare, approaching diminishing returns vs full fine-tune.

For multi-LoRA serving, use the lowest rank that produces acceptable quality. Higher ranks compound memory cost across many active LoRAs.

LoRA adapter loading strategies

In production:

Pre-load all adapters at startup. Simple, predictable. Doesn't scale beyond ~1000 adapters per replica due to memory.
On-demand loading. First request for a new adapter loads it (50-100ms latency hit). Subsequent requests are fast. Good for long-tail adapter usage.
Disk-cached LoRAs. Adapter weights on local NVMe; load on demand. Balances memory and load latency.

vLLM's default is on-demand loading. SGLang offers both. For production multi-tenant serving with many adapters, on-demand is usually right.

Scheduling, admission control, and priority

The scheduler decides, at every iteration: which queued requests to admit, which in-flight sequences to step, which sequences to preempt when KV is full.

vLLM's default

vLLM uses FCFS (first-come-first-serve) with KV-availability constraints. Simple, fair, no priority concept.

Where simple scheduling falls down

Mixed latency targets: chat (200ms) and batch (1-hour) on same replica. FCFS gives them equal priority.
Long-tail outputs: a 10k-token request shouldn't block 100 short ones.
Multi-tenant fairness: tenant A with 100 active requests shouldn't crowd out tenant B's 1.

Beyond FCFS

Production deployments layer scheduling above the inference engine:

Priority queues at the API gateway. Tag requests by priority. Throttle low-priority traffic when high-priority is loaded.
Per-tenant quotas. Token-bucket rate limits per tenant.
Output-length-based scheduling. Preempt requests with high max_tokens first when the cache fills.

vLLM and SGLang both have priority scheduling support. For production, building this at the API gateway is more flexible.

Admission control

The "do I admit now or queue" decision matters for latency tails. Conservative admission keeps the in-flight set small, lowering tail latency at the cost of throughput.

The tuning knob: max_num_seqs. Lower = lower tails, lower throughput. Higher = higher throughput, fatter tails.

A common pattern: set max_num_seqs based on your P95 latency budget. Measure: at max_num_seqs=N, what's P95 first-token latency? Bump N until P95 hits SLO; stop there.

Preemption

When new request arrives and KV is full, somebody gives. vLLM evicts the lowest-priority sequence (longest-running by default). SGLang's RadixAttention often dodges this. TRT-LLM has swap-based preemption.

For most workloads, default eviction is right. Don't tune unless eviction-rate symptoms.

Multi-GPU: TP, PP, EP, DP combinations

When the model doesn't fit on one GPU, or you want more capacity, you scale across GPUs. Four primary strategies, often combined.

Tensor parallelism (TP)

Split each weight matrix across GPUs. Forward pass requires all-reduce after each layer. Standard for models that don't fit on one GPU.

For Llama-3 70B BF16 (140 GB): TP=2 on 2× H100 fits each shard at 70 GB. TP=4 fits at 35 GB. TP=4 also linearly drops per-GPU KV.

The cost: NCCL all-reduce per layer. On NVLink (within a node) cheap. Across nodes (InfiniBand/RoCE) expensive — TP rarely scales past 8.

Pipeline parallelism (PP)

Split the model by layer across GPUs. Token at position N flows GPU 0 → GPU 1 → ... → GPU N-1.

PP introduces "pipeline bubbles" — periods where some GPUs idle. Modern stacks use micro-batching and 1F1B scheduling.

PP is mostly used in training. For inference, PP's bubbles hurt latency, and TP usually wins. Some stacks (TRT-LLM) support PP for very large models exceeding one node's TP capacity.

Expert parallelism (EP)

For MoE models. Distribute experts across GPUs. Each token routes to assigned experts via all-to-all.

KV cache is per layer, not per expert, so EP doesn't change KV. EP is purely MLP routing.

For Mixtral 8×22B on 8× H100 with EP=8: each GPU holds 1/8 of experts. Inter-GPU all-to-all per layer adds overhead, but the compute savings (~21B active out of 141B) more than compensate.

Data parallelism (DP)

Replicate the entire model on each GPU. Each GPU serves independent requests. Simplest scaling.

DP is replication, not parallelism. Useful for scaling out beyond what TP/PP can fit. Most production deployments combine DP with TP: e.g., 4 replicas each with TP=2 across 8 H100s.

Combining strategies

For 8 H100s serving Llama-3 70B:

DP=4 × TP=2: 4 replicas, each 2-GPU. Highest throughput for short-context.
DP=2 × TP=4: 2 replicas, each 4-GPU. Better latency per replica.
DP=1 × TP=8: 1 replica spanning all 8. Maximum capacity per request.

Pick based on concurrency and context-length distribution.

Configuring multi-GPU

vLLM:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1

For DP, run multiple processes (one per replica) behind a load balancer.

For mixed TP+EP on MoE:

vllm serve mistralai/Mixtral-8x22B-Instruct-v0.1 \
  --tensor-parallel-size 4 \
  --expert-parallel-size 2

The major stacks compared

vLLM

The most popular, originated PagedAttention. Default safe choice.

Strengths: huge community, broad model support, well-documented
Weaknesses: not the fastest single-stream throughput; some advanced features lag TRT-LLM
Pick if: starting fresh, multi-tenant production, no specific reason to deviate

SGLang

Built on PagedAttention, extends with RadixAttention.

Strengths: best prefix sharing, excellent for chat with shared prompts, strong agentic support
Weaknesses: smaller community, some operational rough edges
Pick if: chat-heavy with shared prefixes, agent/tool workloads, structured output

TensorRT-LLM

NVIDIA's first-party engine. Fastest on H100/H200/B200.

Strengths: highest peak throughput on NVIDIA, best FP8/FP4, official backing
Weaknesses: locked to NVIDIA, build-time engine compilation complex
Pick if: NVIDIA-only stack, single-tenant max throughput

TGI (Hugging Face)

HF's serving stack, powering Inference Endpoints.

Strengths: tight HF integration, mature
Weaknesses: not always at parity on cutting-edge features
Pick if: deploying via HF Inference Endpoints

LMDeploy

Chinese-developed, strong on Chinese open-weight models.

Strengths: best Qwen/DeepSeek/ChatGLM optimization
Weaknesses: smaller ecosystem outside China, less English docs
Pick if: serving Chinese open-weight models at scale

llama.cpp

CPU-first, GGUF format, runs anywhere.

Strengths: runs on Apple Silicon, AMD, CPU-only, edge
Weaknesses: not competitive for multi-tenant production
Pick if: local/single-user, weird hardware, edge

Decision matrix

Workload	Recommended stack
Multi-tenant chat with shared prompts	SGLang
Multi-tenant general	vLLM
Single-tenant max throughput on NVIDIA	TRT-LLM
HF Inference Endpoints	TGI
Chinese open-weight models at scale	LMDeploy
Local development, demos	llama.cpp or Ollama
Edge deployment	llama.cpp
Apple Silicon	MLX or llama.cpp
AMD MI300/MI350	vLLM (best AMD support)

Comparative benchmarks (mid-2026)

Numbers below are typical sustained throughput for Llama-3 70B on indicated hardware. Exact numbers vary by version; treat as orders of magnitude.

Single H100 SXM 80GB, FP8 weights + FP8 KV, 4k context, no shared prefix:

Stack	Tokens/sec	Notes
vLLM 0.6+	850	Default safe choice.
SGLang 0.4+	880	Slight edge from RadixAttention overhead.
TRT-LLM 0.13+	1100	Custom engine, highest single-stream.
LMDeploy 0.6+	920	Solid all-rounder.

Single H200 SXM 141GB, same configuration:

Stack	Tokens/sec	Notes
vLLM	1100	+30% from H200's bandwidth.
SGLang	1150
TRT-LLM	1450
LMDeploy	1180

8× H100 SXM (TP=4, DP=2), Llama-3 70B FP8, 32k context, 50 concurrent users:

Stack	Tokens/sec aggregate	P95 TTFT
vLLM	4200	1.8s
SGLang	4500	1.6s
TRT-LLM	5100	1.4s
LMDeploy	4300	1.7s

8× H100 SXM, chat workload with 1k shared system prompt, 100 concurrent users, 4k context:

Stack	Tokens/sec aggregate	Prefix hit rate
vLLM 0.6+ (block-level)	3800	91%
SGLang (RadixAttention)	5100	97%
TRT-LLM (built-in cache)	4200	89%

SGLang's RadixAttention pulls ahead substantially on prefix-shared workloads. For non-shared workloads, the stacks are within 10% of each other.

Stack version stability

Versions matter. Highlights from 2024-2026 changelogs:

vLLM 0.6 → 0.7: prefix caching default-on, multi-step async scheduling, EAGLE auto-tuning.
SGLang 0.3 → 0.4: structured output performance, SGLang language extensions.
TRT-LLM 0.10 → 0.13: FP4 native support on Blackwell, paged attention overhauls.

Pin to a known-stable version in production. Don't auto-update.

vLLM internals: how it actually works

A quick architectural sketch of vLLM helps when debugging. The major components:

Engine (LLMEngine): the orchestrator. Owns the model, KV cache pool, scheduler, and tokenizer.

Scheduler (Scheduler): per-iteration decision maker. Decides which sequences to admit, which to preempt. State machine over WAITING / RUNNING / SWAPPED queues.

Block manager (BlockManager): manages the physical KV cache pool. Allocates blocks to sequences, tracks which blocks are free, handles prefix-caching's hash-based block lookup.

Worker (Worker): per-GPU process that holds model weights and executes forward passes. Workers communicate via NCCL for TP/PP.

ModelRunner: wraps PyTorch model code, handles batching, and integrates with the paged-attention kernels.

API server (vllm.entrypoints.openai): HTTP server exposing OpenAI-compatible API. Translates HTTP requests into engine calls.

The flow for a single request:

HTTP request hits API server.
Tokenized; converted to a SequenceGroup object.
Submitted to the engine.
Engine adds it to the WAITING queue.
Each scheduling iteration:
- Scheduler decides if any WAITING sequences fit (KV pool has space).
- Admits some WAITING sequences (state → RUNNING).
- Preempts RUNNING sequences if needed (state → SWAPPED or back to WAITING).
- ModelRunner executes one forward pass on all RUNNING sequences.
- Sequences advance one token; some finish.
Tokens stream back through the API server to the client.

Knowing this helps with debugging:

Stuck request? Probably in WAITING because KV is full. Reduce max_num_seqs or add capacity.
Slow first token? Long queue time, or large prefill blocking the scheduler.
Inconsistent throughput? Scheduler thrashing — admitting and preempting in tight loops.

vLLM's continuous-batching internals

The scheduler's batch construction logic:

def _schedule_running(self):
    # Sequences that are mid-decode
    running = self.running
    blocks_to_swap_in = []
    blocks_to_swap_out = []
    
    while running:
        seq = running.peek()
        if not self.block_manager.can_append(seq):
            # Need to preempt something
            victim = running.pop_lowest_priority()
            self._preempt(victim, blocks_to_swap_out)
        else:
            self.block_manager.append_slot(seq)
            running.pop()
    
    return BatchedSequences(running, blocks_to_swap_in, blocks_to_swap_out)

The can_append check is what causes "OOM but not really" symptoms when the cache is fragmenting. With paged-attention, fragmentation is bounded but not zero.

Async multi-step scheduling

vLLM 0.7+ introduced async multi-step scheduling. Instead of one Python scheduling decision per token, the scheduler plans 4-8 steps ahead and dispatches them as a batch to the model. Reduces Python overhead — a real bottleneck in earlier versions where the GPU could outpace Python's per-step decision making.

Concrete improvement: ~15-30% throughput win on small-batch high-frequency workloads (chat with short responses).

Configurable via --num-scheduler-steps 8 in vLLM 0.7+.

SGLang internals

SGLang's architecture differs from vLLM in important ways:

RadixAttention is the centerpiece. Where vLLM uses a hash table for prefix-block lookup, SGLang maintains a radix tree of token sequences. The tree is keyed by token IDs; each node represents a unique prefix.

When a new request arrives:

SGLang traverses the tree from the root, matching the request's prefix tokens.
The longest matching node becomes the request's "mount point."
Only the suffix (tokens beyond the matched prefix) requires KV computation.
After completion, the request's leaf is added to the tree (potentially evicting older leaves).

This generalizes vLLM's block-level sharing to any shared prefix length. Where vLLM shares whole blocks (16 tokens), SGLang shares at the token level.

Frontend language: SGLang ships a Python DSL (sgl.gen, sgl.fork, etc.) for expressing common patterns: parallel generation, structured output, branching dialogues. The DSL compiles to efficient batched inference.

@sgl.function
def multi_turn(s, question):
    s += "Question: " + question + "\n"
    s += "Answer: " + sgl.gen("answer", max_tokens=200)
    s += "Follow-up: " + sgl.gen("followup", max_tokens=100)

Under the hood, SGLang batches the multiple generations within one request, sharing the prefix automatically.

Constrained generation: SGLang's structured output uses logit masking to constrain generation to a regex or grammar. Tightly integrated with RadixAttention — the constraint state is part of the radix tree.

For chat-heavy workloads with shared system prompts, SGLang's RadixAttention often delivers 2-3× the throughput of vLLM's block-level prefix caching. For workloads without prefix sharing, the two are within 10%.

TensorRT-LLM internals

TRT-LLM works differently from vLLM/SGLang:

Engine compilation: instead of running PyTorch at inference time, TRT-LLM compiles your model into a custom CUDA engine ahead of time. Compilation happens once (takes 5-30 minutes); inference runs the compiled engine.

trtllm-build --checkpoint_dir /path/to/llama-3-70b \
  --gpt_attention_plugin float16 \
  --gemm_plugin float16 \
  --use_fp8_context_fmha enable \
  --max_batch_size 32 \
  --max_input_len 8192 \
  --max_output_len 2048

Pros:

Highest single-stream throughput on NVIDIA. ~30-40% faster than vLLM on Llama-3 70B.
Deeply integrated with NVIDIA's hardware (Hopper FP8, Blackwell FP4).
Production-tested at scale (NVIDIA's own NIM uses it).

Cons:

Engine compilation is opaque — debugging is harder.
Fixed batch size and context length at compile time. If your workload mix changes, you may need to recompile.
Smaller community than vLLM.

Triton Inference Server typically wraps TRT-LLM engines for HTTP serving. It provides the OpenAI-compatible API layer. Together, the stack is "Triton + TRT-LLM."

Continuous batching: TRT-LLM has its own implementation, often called "in-flight batching" in NVIDIA docs. Functionally equivalent to vLLM/SGLang but with NVIDIA-internal optimizations.

Paged KV: native support via paged-attention plugins. Same concept as vLLM, NVIDIA implementation.

When to pick TRT-LLM:

Single tenant, single primary model, scale enough to amortize compilation overhead.
Locked to NVIDIA hardware.
Maximum throughput is a key metric.

For most teams: vLLM is easier to operate. TRT-LLM is the right answer for hyperscale single-tenant production.

A typical production deployment architecture

For a serious production setup:

              [Cloudflare / CDN]
                      ↓
              [Application LB]
                ↓        ↓
            [API GW]   [API GW]   ← rate limiting, auth, priority
              ↓ ↓ ↓ ↓
        [Replicas R1...Rn]      ← vLLM/SGLang/TRT-LLM, autoscaled
              ↓
        [Shared model storage]   ← S3/GCS for model weights
              ↓
        [Observability]          ← Prometheus, Grafana, traces

Components:

CDN: terminates TLS, caches static assets. Doesn't directly proxy LLM traffic but handles surrounding services.
Application load balancer: routes by URL path, handles cookies/headers.
API gateway: authentication, rate limiting, priority queuing, optional response caching for deterministic queries.
Replicas: stateless inference replicas. Each is a single instance of vLLM/SGLang/TRT-LLM.
Shared model storage: S3/GCS with weights. Replicas pull at startup. Common pattern: bake into container for fast cold-start.
Observability: metrics from each replica aggregated centrally.

This pattern is similar to any HTTP microservice; the LLM-specific bits are the replicas themselves.

Latency engineering: prefill, decode, tails

Latency in LLM serving is multi-dimensional.

The metrics that matter

Time to first token (TTFT): from receipt to first output. Dominated by prefill. The metric users feel.
Inter-token latency (ITL): time between consecutive tokens. Dominated by decode. Streaming smoothness.
End-to-end latency: TTFT + (output_tokens × ITL). For batch jobs.
Tokens per second (per request): 1 / ITL.
Aggregate throughput: total output tokens/sec across all requests.

These can move in opposite directions. Optimizing aggregate throughput often hurts P99 TTFT.

Reducing TTFT

TTFT = prefill cost + queue wait.

Reduce prompt length (prompt compression, smarter retrieval).
Enable prefix caching — cached prefixes skip prefill.
Reduce queue wait — lower max_num_seqs at throughput cost.
Use chunked prefill — interleave prefill chunks with decode.
Faster hardware: H200 prefills ~1.3× faster than H100, B200 ~2× faster.
TP=4 vs TP=2: more compute parallelism reduces prefill latency.

Reducing ITL

ITL is per-decode-step time, dominated by KV cache reads.

Higher HBM bandwidth GPU: H200 has 4.8 TB/s vs H100's 3.0 TB/s.
Quantize the KV (FP8 or INT4): half/quarter bytes per step.
Speculative decoding: 2–3× effective ITL on suitable workloads.
Fewer concurrent requests: each in-flight request adds compute.

Managing tail latency

P99 latency is often 5–10× P50. Sources:

Long-context requests blocking the batch: chunked prefill helps.
Eviction events: avoid by sizing KV with headroom.
Cold starts: warm up explicitly.
NCCL collective hiccups: reduce TP if you can.
Garbage collection (Python): tune Python GC settings.
Preemption from new arrivals: trade against throughput.

A practical rule: aim for P99/P50 < 4×.

SLO budgets for common applications

Interactive chat: P95 TTFT < 1s, P95 ITL < 50ms.
Code completion: P95 TTFT < 200ms.
Agent tool calls: P95 TTFT < 500ms, P95 end-to-end < 5s.
Search/RAG answers: P95 TTFT < 2s, P95 ITL < 80ms.
Batch document processing: P99 end-to-end < 60s.

Profiling latency in production

When latency is wrong, the question is which phase is slow. Tools:

NVIDIA Nsight Systems: per-GPU timeline showing every CUDA kernel and NCCL collective. Run for 10 seconds during a representative load:

nsys profile --trace=cuda,nvtx \
  --output=trace.qdrep \
  python serve.py

Open trace.qdrep in Nsight UI. Look for:

Long single kernels (custom op without good kernel).
NCCL collectives taking longer than expected (network issue).
Gaps between kernels (CPU-GPU sync overhead).

Stack-level metrics (Prometheus):

vllm:time_to_first_token_seconds (histogram)
vllm:time_per_output_token_seconds
vllm:request_queue_time_seconds
vllm:e2e_request_latency_seconds

Bucket by request size to identify which workload class is causing tails.

Application-level traces (OpenTelemetry): trace spans for tokenization, queue wait, prefill, decode, network return. Identifies the slow phase per request.

Latency-affecting hyperparameters

Some configurations heavily impact latency:

Parameter	Effect on latency	Trade-off
`max_num_seqs`	Lower = lower TTFT, lower throughput	Linear
`max_num_batched_tokens`	Higher = better throughput, longer step time	Linear
`enable_chunked_prefill`	Smoother latency under long prefills	Slight efficiency loss
`block_size`	Larger = better kernel efficiency, more tail waste	Modest
TP degree	Higher = lower TTFT for prefill, more comm overhead	Asymptotic
KV format	Smaller = lower ITL (less data per step)	Quality cost

Streaming latency mechanics

In streaming mode, ITL determines user-perceived smoothness. Tips:

Flush after every token, not every batch. Adding 50ms of buffering halves perceived smoothness.
Avoid heavy post-processing on the response path. Token streaming should be raw; transformations happen client-side.
Server-Sent Events with proper Connection: keep-alive and Cache-Control: no-cache headers.
Client-side: render incoming tokens as they arrive. Don't wait for sentence boundaries.

The user's perception of "fast" is mostly about TTFT (when did the model start responding) and steadiness (no long pauses mid-response). ITL of 30-50ms feels instant. Above 100ms feels laggy.

Capacity planning

How many GPUs do you need? The math.

Inputs

Model: weight memory + KV-per-token.
Workload: peak concurrent users, average context length, average output length.
Latency SLO.
Hardware.

Procedure

Pick weight quantization. Compute total weight memory.
Pick TP degree. Smallest that fits weights per GPU with ~30 GB headroom.
Compute per-GPU KV per token at chosen TP and KV format.
Compute KV memory budget per GPU = total HBM − weights/TP − headroom.
Max concurrent requests at target context = KV budget / per-request KV.
Apply prefix caching multiplier (1.0 to 5×).
Replicate to handle concurrent users.
Validate latency.

Worked example: chat at scale

100 peak concurrent users. Llama-3 70B. 4k context, 500 output. SLO: P95 TTFT < 1s.

FP8 weights: 70 GB.
TP=2 → 35 GB per GPU.
Per-GPU KV: 80 KB/token.
4k context = 320 MB per request.
KV budget per GPU: 80 - 35 - 30 = 15 GB. Max ~46 concurrent.
Prefix hit ~95% → 4× effective: 184 concurrent per replica.
100/184 = 1 replica (use 2 for failover).
P95 TTFT at 4k: ~150ms. Met.

Result: 2× H100 + TP=2 + FP8 + prefix caching handles 100 chat users.

Worked example: long-context RAG

20 peak concurrent. Llama-3 70B. 32k context, 500 output. SLO: P95 TTFT < 3s.

FP8: 70 GB.
TP=2 (35 GB per GPU).
Per-GPU KV: 80 KB/token.
32k context = 2.56 GB per request.
KV budget per GPU: 15 GB. Max ~5 concurrent.
RAG prefix hit ~20%: ~6 effective.
20/6 = 4 replicas. 8× H100 total.
P95 TTFT at 32k: ~1.4s prefill + 0.4s queue = 1.8s. Met.

Result: 8× H100 across 4 TP=2 replicas handles 20 RAG users.

Worked example: agentic workload with high concurrency

200 peak concurrent users running agents. Llama-3 70B. Average input 2k tokens (system prompt + recent conversation), average output 1500 tokens (long thinking). SLO: P95 TTFT < 800ms, P95 end-to-end < 30s.

FP8 weights: 70 GB. TP=2 → 35 GB per GPU.
KV per request: (2k + 1500) × 80 KB = 280 MB per GPU.
KV budget per GPU at TP=2: 15 GB. Max ~50 concurrent.
Prefix caching ~80% (system prompt shared): 1.6× effective → 80 concurrent per replica.
200 / 80 = 3 replicas. 6× H100 with TP=2.
Speculative decoding (EAGLE-2): ~2.2× throughput on agentic workloads. Effectively shrinks decode time.
Validate: agentic prefill is short (2k); ~150ms. P95 TTFT met. End-to-end at 1500 output × 12ms/token (with spec-decode) = 18s. Met.

Result: 6× H100, TP=2, 3 replicas, FP8, prefix caching, EAGLE-2 spec-decode. ~$24/hr. Serves 200 concurrent agentic users.

Worked example: very-long-context document processing

10 peak concurrent users. Llama-3 70B. Average input 200k tokens (whole legal documents), output 5k tokens. SLO: P95 end-to-end < 90s.

FP8 weights: 70 GB. TP=4 → 17.5 GB per GPU on H100.
KV per request at TP=4: 40 KB/token. 200k context = 8 GB per request.
KV budget per GPU at TP=4: 80 - 17.5 - 30 = 32.5 GB. Max ~4 concurrent.
No prefix sharing (each document unique). 1× multiplier.
10 / 4 = 3 replicas. 12× H100 needed.
P95 prefill at 200k: ~12 seconds. P95 decode at 5k tokens: ~50 seconds. Total ~62s. Met.

Result: 12× H100 across 3 TP=4 replicas. ~$48/hr. Serves 10 concurrent long-context users.

The pattern: long context demands higher TP and accepts lower throughput. Replicas scale concurrency, not per-request capability.

Cost economics

What does serving actually cost?

Indicative numbers (mid-2026)

Llama-3 70B FP8 on 2× H100 ($4/hr lease):

~1500 tok/s aggregate
5.4M tokens/hour
$0.74/M output tokens

DeepSeek-V3 MLA on 8× H200 ($24/hr):

~3000 tok/s aggregate
10.8M tokens/hour
$2.22/M tokens

Compare to APIs:

OpenAI GPT-4o-mini output: ~$0.60/M
Anthropic Claude Sonnet output: ~$15/M
DeepSeek API: ~$1.10/M

Optimization wins

FP8 KV: 2× capacity, 0.1 quality cost. Cuts cost ~half if KV-bound.
Prefix caching: 2-5× capacity on shared prefixes.
Speculative decoding: 2-3× decode throughput.
Right-sized hardware.

When self-hosting beats API

Below 10M tokens/month: API. Above 100M/month: self-host. Between: depends on workload shape.

Detailed cost analysis: 1B tokens/month

Take a serving requirement: 1B output tokens/month. Compare API vs self-host.

API (OpenAI gpt-4o-mini at $0.60/M output):

1B × $0.60/M = $600/month.
Plus input tokens: at 4k input × 1B output / 200 output per request = 5B input tokens. At $0.15/M = $750/month.
Total: ~$1,350/month.

Self-hosted (Llama-3 70B FP8 on 2× H100):

1B output tokens at 1500 tok/sec aggregate = 1B / 1500 / 3600 = 185 hours of compute per month.
2× H100 lease at $4/GPU-hr = $8/hr.
Active compute cost: 185 × $8 = $1,480.
Plus 24/7 idle baseline (assuming 50% utilization): $8 × 24 × 30 × 0.5 = $2,880/month total cost.
Plus engineering, monitoring, on-call.

For 1B tokens/month, API is competitive on raw cost and dramatically cheaper on operational overhead. Self-hosting makes sense at 5B+ tokens/month or when your model isn't available via API.

Cost optimization checklist

If serving cost is a concern:

Quantize weights: FP8 saves 50% memory, often translates to fewer/smaller GPUs.
Quantize KV: FP8 KV halves KV memory.
Enable prefix caching: 2-5× throughput on shared-prefix workloads.
Speculative decoding: 2-3× decode throughput on agentic workloads.
Right-size hardware: don't run on B200 if H100 suffices.
Spot/preemptible instances: 50-70% off for batch workloads.
Multi-LoRA: consolidate fine-tunes onto fewer base-model replicas.
Disaggregated prefill+decode: ~30% savings for skewed prefill:decode ratios.

Stack these. The compounding can be 5-10× over a naive deployment.

Autoscaling and traffic shaping

Production traffic isn't steady. Handling bursts without overspending is its own discipline.

Why LLM autoscaling is hard

GPUs are slow to start. Cold-start a Llama-3 70B replica: 60-180 seconds.
GPUs are expensive. Spinning up for a brief burst costs more than absorbing latency.
Capacity isn't fungible. A 32k-context request can't go to a 4k-max replica.

Patterns that work

Pre-warmed pools. Keep small pool of warm replicas. Scale up via warming during expected peaks.
Burst into cheap inference. Primary on dedicated GPUs, fallback on cheaper hardware.
Backpressure at the API gateway. Reject excess at the edge.
Spot instance fleets for batch workloads.

Concrete autoscaling parameters

For Kubernetes HPA:

Scale up trigger: P95 TTFT > 1.5× SLO sustained 60s.
Scale up step: +25%, capped at 4 per event.
Scale up cooldown: 5 min.
Scale down trigger: GPU utilization < 40% sustained 15 min.
Scale down step: -1 replica.
Scale down cooldown: 30 min.

Multi-region deployment patterns

For globally-distributed users:

Active-active independent: each region has full capacity for global traffic. DNS routes by geography. Failover is automatic but cold-start during failover takes 60-180s.

Pros: best baseline latency, geographic data sovereignty. Cons: 2-3× cost (paying for 100% capacity in each region for failover headroom).

Active-active partitioned: each region serves a portion of traffic permanently. No failover; if a region dies, its traffic is denied or routed at higher latency.

Pros: cost-efficient, predictable. Cons: regional outages cause real user impact.

Active-passive: primary region serves all traffic; secondary stays warm but idle. On primary failure, DNS shifts traffic to secondary.

Pros: simplest. Capacity in primary region only (until failover). Cons: failover latency is brutal — 60-180s of degraded service while DNS propagates.

For most LLM serving, active-active independent is the right choice when global users have tight latency SLAs. Active-active partitioned works for cost-sensitive deployments. Active-passive is rarely the right answer for user-facing services.

Cross-region prefix caching

A subtle issue with multi-region: each region has its own prefix cache. The same shared system prompt has to be cached separately in each region.

For multi-tenant deployments with stable prefixes, this is acceptable — the warm-up cost amortizes quickly. For very-bursty workloads with brief prefix overlap, the per-region cache miss can hurt.

Some experimental setups distribute prefix cache state across regions. Not yet standard.

Case study: a real production deployment

A composite based on common patterns observed in 2025-2026 deployments.

Setup: SaaS company. 50M active users globally. Chat-heavy workload (95% chat, 5% RAG). Average request: 800-token input, 300-token output.

Hardware: 200 H100s across 3 regions (us-east, us-west, eu-central). 80 H100s us-east, 80 us-west, 40 eu-central. TP=2 across, ~25 replicas of TP=2 each.

Stack: SGLang. RadixAttention chosen specifically for the shared system prompt.

Configuration:

--kv-cache-dtype fp8_e4m3 --calculate-kv-scales
--enable-prefix-caching (default in SGLang)
--max-num-seqs 96
EAGLE-2 speculative decoding enabled

Metrics in production:

Aggregate throughput: ~50,000 tokens/sec across all regions.
P95 TTFT: 950ms (target 1s).
P95 ITL: 35ms (target 50ms).
Prefix cache hit rate: 91%.
KV utilization per replica: 65-75%.
GPU utilization: 78%.
Cost: ~$160k/month.

Observed wins:

Switched from vLLM to SGLang in Q3 2025: 1.7× throughput improvement on shared-prefix workload. Saved ~$100k/month.
Added EAGLE-2 spec-decode: another 30% throughput gain.
FP8 KV: 2× capacity per replica, halved replica count for the same workload.

Operational challenges:

Random NCCL hangs roughly once per month per replica. Mitigated with NCCL_TIMEOUT=600 and automatic restart.
Cross-region sync of LoRA adapters is a pain. Solved by baking adapters into container images.
Prefix cache invalidation on tokenizer changes caused output corruption once. Now: clear cache on every deploy, automated.

The lessons: stack choice mattered (SGLang's prefix sharing was a real win for this workload); operational discipline mattered (NCCL_TIMEOUT, deploy hygiene); quality monitoring caught a tokenizer issue that pure throughput metrics missed.

Observability and SLO design

You can't optimize what you don't measure.

Core metrics

Metric	What it tells you
Active requests	Concurrency.
Queued requests	Backpressure.
KV utilization	Memory pressure.
Eviction rate	Saturation.
Prefix cache hit rate	Efficiency.
TTFT (P50/P95/P99)	User-felt latency.
ITL (P50/P95)	Streaming smoothness.
Tokens-per-second	Revenue indicator.
GPU utilization	Hardware efficiency.
Error rate	Reliability.

Useful SLOs

For interactive chat:

TTFT P95 < 1s
ITL P95 < 50ms
End-to-end P95 < 5s
Error rate < 0.1%
Availability 99.9%

For agent workloads:

TTFT P95 < 2s
End-to-end P95 < 30s

Alerts that matter

TTFT P95 > SLO for 5 min
Eviction rate > 5/sec for 5 min
GPU memory free < 5 GB
Error rate > 1% for 1 min
Replica count below minimum
KV utilization > 95% for 2 min

Keep alert volume low.

A reference Grafana dashboard

The dashboards that matter for production LLM serving:

Top row — health at a glance:

Aggregate tokens-per-second (across all replicas).
TTFT P50/P95/P99 (single graph, multiple lines).
Error rate (per replica).
Active concurrent requests.

Second row — scaling signals:

KV utilization per replica.
Eviction rate per replica.
GPU memory free per replica.
Replica count vs autoscale target.

Third row — workload breakdown:

TTFT histogram bucketed by context length (4k/16k/64k/128k+).
Tokens-per-second by tenant (if multi-tenant).
Prefix cache hit rate.

Fourth row — anomalies:

Slow-request distribution (P99/P50 ratio over time).
Error rate breakdown by HTTP code.
Request rejection rate.

This is enough to operate at scale. Add latency-by-tenant if you have SLA breakdowns.

Tracing in distributed deployments

For multi-replica deployments, add tracing:

API gateway emits trace ID.
Inference engine includes trace ID in logs.
Application client correlates by trace ID.

OpenTelemetry is the standard. Most cloud-native stacks integrate with Jaeger, Tempo, or vendor-specific tracing.

When debugging a slow request: pull the trace, see exactly where time was spent (queue, prefill, decode, network return). Beats grep'ing logs.

Streaming, tool use, structured output

Streaming (SSE)

OpenAI-compatible API uses data: prefixed JSON chunks. Tips:

Flush after every token.
Keepalives every 15s during long thinking phases.
Send finish_reason at the end.
Handle disconnects gracefully — abort to free KV.

Tool use / function calling

Stacks that constrain output to tool-call schema: vLLM (Outlines/LMFE), SGLang (native), TGI (guidance), TRT-LLM (NIM extensions).

KV implication: tool calls have predictable structure, so prefix caching is high. Speculative decoding works extremely well.

Structured output (JSON, regex)

Outlines: grammar-constrained generation, masks logits at each step.
LMFE: similar.
SGLang's structured output: built-in, optimized.

Quality note: constrained generation can hurt model quality in edge cases. For schema-strict APIs, this is what you want; for "nudge toward JSON," prompt and parse with retries.

Streaming implementation details

The Server-Sent Events (SSE) wire format that the OpenAI API uses:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" world"}}]}

data: [DONE]

Each event is data: <JSON>\n\n. The terminator is data: [DONE].

Implementation gotchas in production:

Buffering: most HTTP libraries and proxies buffer responses. To stream, you have to explicitly disable buffering (X-Accel-Buffering: no for nginx; framework-specific for others). If users see chunks arrive in big bursts, this is the cause.

Keepalive: long thinking phases (10-30s of internal reasoning before the first visible token) can trigger proxy timeouts. Send a comment line (: keepalive) every 15 seconds during quiet periods.

Chunked transfer encoding: required for streaming. Some proxies or load balancers will buffer until the full response, defeating streaming. Verify with curl --no-buffer.

Client disconnects: when a user closes their browser, the connection drops mid-stream. The serving stack should detect this (broken pipe on write) and abort the request to free KV.

async def stream_response(request, generator):
    try:
        async for token in generator:
            await request.send(f"data: {json.dumps(token)}\n\n")
        await request.send("data: [DONE]\n\n")
    except (ConnectionResetError, BrokenPipeError):
        generator.abort()  # Free KV on the inference engine
        raise

Backpressure: if the client reads slowly (e.g., mobile network), the serving stack's send buffer fills. Some implementations block; others drop tokens. Production stacks should bound the buffer and disconnect on prolonged backpressure.

Tool calling implementation

Tool calls in OpenAI format:

{
  "model": "...",
  "messages": [{"role": "user", "content": "What's the weather in Paris?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {...}}}]
}

The model responds with either a text response or a tool call:

{
  "choices": [{
    "message": {
      "tool_calls": [{
        "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
      }]
    }
  }]
}

Implementation: the serving stack constrains generation to either output text or output a tool-call structure matching the schema. Stacks like SGLang and vLLM (with Outlines) constrain at the logits level, guaranteeing structurally valid output.

For multi-turn tool use, the application:

Sends user message + tool definitions.
Receives tool_call response.
Executes the tool externally.
Appends tool result to conversation; calls model again.
Repeats until model produces text response.

Each round is a separate inference request. KV cache from prior rounds is reused via prefix caching automatically — the conversation history is the prefix.

Streaming tool calls

A subtle complication: streaming + tool calls. Most stacks stream tool-call generation token-by-token like text. The client has to parse partial JSON. The OpenAI API standardizes a delta format that includes partial tool calls:

data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"{\"city\":"}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":" \"Paris\"}"}}]}}]}

Clients accumulate the partial arguments; when generation completes, the full tool call is reconstructed. Most LLM SDKs handle this for you.

Operational runbook

A condensed playbook for operating LLM serving at scale.

Daily checks

Review TTFT P95 and P99 for the last 24h. Compare to baseline.
Review error rate. Investigate any tenant with >0.5% errors.
Spot-check GPU utilization. If consistently <70%, you're over-provisioned.
Spot-check KV utilization. If consistently <50%, you're over-provisioned (or KV format is too generous).
Check eviction rate. Should be near zero in healthy operation.

Weekly checks

Run nccl-tests to validate fabric health hasn't regressed.
Review autoscaling decisions. Were any scale-up events delayed? Any scale-downs caused user-visible latency?
Update model dependencies. Are you on the latest vLLM/SGLang patch version?
Review cost report. Any anomalies?

Monthly checks

Run a representative load test against your stack. Compare to last month's numbers.
Review observability dashboards. Are the right metrics being tracked? Any alerts that fire too often or never fire?
Check for stack version updates. Is there a major version with relevant features?

Incident response: TTFT spike

Symptoms: P95 TTFT jumps from 800ms to 3s. P99 worse.

Triage:

Look at GPU utilization. If high, you're at capacity → scale up or shed load.
Look at queue depth. If large, scheduler is admitting too few requests → tune max_num_seqs.
Look at prefill latency by context bucket. If long-context bucket is slow, a single 200k-token request might be blocking the batch.
Look at NCCL performance via Nsight. If collectives are slow, network issue.

Incident response: error rate spike

Symptoms: error rate jumps from 0.1% to 5%.

Triage:

Check error breakdown by HTTP code. 503 = overloaded; 500 = internal; 429 = rate limited.
Spot-check error logs. Are they uniform (one bug) or varied (multiple causes)?
Check replica health. Any replicas in unhealthy state?
If rolling back fixes it, the most recent deploy is the culprit.

Incident response: a replica is OOM'ing

Symptoms: replica crashes; pod restarts; back to OOM in 10 minutes.

Triage:

Look at memory growth pattern. Step-function (large request) vs gradual (memory leak).
If step-function: a single request exceeded --max-model-len. Tighten the limit.
If gradual: stack version may have a known leak. Check changelogs.
Reduce max_num_seqs as immediate mitigation.

Incident response: cascading failure

Symptoms: one replica goes down; load redistributes; another goes down; cascade.

Mitigations:

Reduce traffic admission rate at the API gateway (preserve remaining capacity).
If serving stack supports it, enable circuit breaker — temporarily reject new requests on overloaded replicas.
Don't restart replicas in parallel; sequential restart is safer.

Capacity planning iteration

Every quarter:

Review actual peak concurrency vs design capacity.
Identify workload changes (new tenants, new models, new SLAs).
Re-run capacity planning math.
Adjust replica count, GPU type, parallelism strategy.

Don't just scale by intuition; the math is the right framework.

Failure modes and incident response

OOM during prefill

200k-token prompt; allocated for 128k. Replica crashes.

Fix: always set --max-model-len, reject at API layer.

NaN propagation from FP8 KV

A single overflow corrupts KV. Output becomes garbage.

Fix: enable --calculate-kv-scales; fall back to BF16 KV if calibration suspect.

Tokenizer / model mismatch

Stale token IDs in cached blocks.

Fix: clear cache on every model deploy.

NCCL hangs on multi-GPU

Specific node has network/driver issue.

Fix: NCCL_TIMEOUT env; restart on collective hangs. Health check via inference probe, not just port check.

Slow disk IO on model load

Cold start 5+ minutes from network share.

Fix: bake models into image, or use local NVMe.

Memory leak

Replica works hours then OOMs.

Causes: pinned-memory leak, allocator fragmentation, Python objects.

Debug: capture growth pattern. Profile with nvidia-smi and stack metrics.

Cascading failures

One replica fails, traffic routes to others, they overload.

Fix: keep headroom (don't run at 95%), graceful degradation, circuit breakers.

The bottom line

The named problems are head-of-line blocking and KV fragmentation: padded static batching strands the GPU on the slowest reply, and contiguous worst-case KV reservations strand half of HBM. The solution is to treat LLM serving like an operating system — preemptive per-step scheduling (continuous batching) and demand-paged KV memory (PagedAttention), composed with prefix caching and speculative decoding on top. The single biggest lever is using a serving stack that does all of this by default; rolling your own loop is the most expensive mistake in production LLMs.

What to do if you take only this away:

Default to vLLM or SGLang. Don't serve LLMs from model.generate() in production, ever.
Measure KV utilization, not just GPU utilization. If KV is under 80%, you have headroom that paging should be capturing.
Turn on prefix caching whenever requests share a system prompt or document context — it is a 2-10x throughput win for chat workloads.
Add speculative decoding only after batching and paging are already saturating compute; it helps decode-bound, low-concurrency traffic the most.
Set SLOs in TTFT and TPOT (time-per-output-token), not end-to-end latency, so prefill and decode tune independently.

Next, read NVIDIA datacenter GPUs for why HBM bandwidth dominates decode throughput, and collective communication for AI training for the tensor-parallel collectives that determine your multi-GPU serving topology.

FAQ

Q: Should I use vLLM or SGLang?

Default vLLM. Switch to SGLang specifically when you have heavy prefix sharing, need their structured-output features, or do agentic workloads.

Q: When does TensorRT-LLM make sense?

NVIDIA-only, one primary model, deploying at scale, specifically need maximum throughput.

Q: How do I handle a 1M-token context?

Hybrid architectures (Jamba), MLA-based models (DeepSeek), aggressive sparse attention, hierarchical KV with NVMe offload, or simply not. Use the architecture, not the optimization.

Q: Should I use disaggregated prefill/decode?

If you have very skewed prefill:decode ratios (RAG with 32k input, 200 output), maybe. ~30% cost saving, operational complexity is the trade.

Q: Why does my P99 TTFT spike randomly?

Long-context requests blocking batch, eviction events, Python GC pauses, NCCL hiccups. Investigate one at a time.

Q: Can I run a quantized model in production?

Yes. FP8 weights + FP8 KV is the modern default on Hopper.

Q: How do I scale serving for a sudden 10× traffic spike?

You don't, fully. GPU cold start is 60-180s. Pre-warmed pool, backpressure, graceful degradation.

Q: How do I serve different fine-tunes efficiently?

Multi-LoRA. One base model, dynamic adapters.

Q: What about CPU inference?

7B-class feasible (10-30 tok/sec on beefy CPU). 70B+ painfully slow. llama.cpp is canonical.

Q: How do I migrate stacks?

In phases. Stand up new stack in parallel, replay recorded trace, compare, ramp.

Q: How do I handle structured output reliably?

Outlines, LMFE, or SGLang's structured output. Constrain at logits level.

Q: What's the cheapest way to serve LLMs at small scale?

< 10M tokens/month: don't self-host. 10-100M: single H100 with 7B-class model + vLLM.

Q: How do I migrate BF16 to FP8?

vLLM and TRT-LLM both support runtime quantization (BF16 → FP8 at load). Quality-test on workload first.

Q: Multiple models on one server?

Process-per-model is standard. Multi-model in one process is experimental.

Q: What happens to tokens already generated when a request times out?

Truncated. Client receives partial (streaming) or no output. Best practice: clients handle partial gracefully.

Q: How does serving change for reasoning models (o1, R1)?

Long internal "thinking" before visible answer. KV grows much more during decode. Size capacity for thinking budget.

Q: What about multimodal (vision) models?

Vision tokens count toward total. 256-1024 tokens per image typical. Plan for vision-token budget.

Q: Should I worry about adversarial prompts?

Yes if multi-tenant. Prompt injection, DoS, data exfiltration. Layer defenses: input validation, output filtering, rate limiting.

Q: Will hosted APIs always be cheaper than self-hosting?

No. Below ~100M tokens/month, APIs win. Above, self-hosting wins for steady traffic.

Q: How do I handle high-priority traffic (e.g., paying customers vs free)?

API gateway-level priority queues are the standard. Tag requests by priority; route high-priority to dedicated replicas or admit them ahead of low-priority. vLLM 0.7+ has a priority field that the scheduler honors; use it.

Q: What happens if a request is sent to a replica with the wrong model?

Modern stacks reject the request with HTTP 404 (model not found). Some stacks have "model auto-loading" that loads the model on demand — this adds 60-180s cold start. Avoid auto-loading in production.

Q: How do I deploy a model update without downtime?

Blue-green deployment. Stand up a new replica pool with the updated model, drain traffic from the old pool, decommission the old pool. Most production stacks support graceful drain via SIGTERM.

Q: What's the right PR/QA process for serving stack changes?

Three stages:

Soak test in dev with realistic load. Watch for memory leaks, error rate spikes.
Canary deploy to a small fraction of production traffic. Compare metrics to baseline.
Ramp to 100% over hours/days. Watch dashboards.

Skip the canary stage at your peril.

Q: How does serving change for very small models (< 1B parameters)?

The dynamics flip. Small models are compute-bound at modest concurrency; KV is irrelevant. Continuous batching and paging matter less. Throughput is determined by raw GEMM speed.

For < 1B models on H100s, you'll see 10,000+ tokens/sec aggregate trivially. The serving stack barely matters; the model is the bottleneck.

Q: How do I serve a model with custom architecture?

Most stacks support vLLM-compatible model definitions. If your architecture is exotic (custom attention, custom MLP), you may need to write integration code. vLLM has the most extension docs.

Q: Should I use INT4 weights for production serving?

Yes if memory-bound and quality cost is acceptable. AWQ INT4 is the most-tested format. Test on your workload — quality drop varies by model and task.

Q: What about MLX or other Apple Silicon stacks?

MLX is the Apple Silicon native choice. Competitive for single-user serving; not for production multi-tenant. llama.cpp also runs well on Apple Silicon.

Q: How does batching affect quality?

It doesn't, with one caveat: numerical determinism. Different batch sizes can produce slightly different outputs due to floating-point reordering. For greedy decoding (temp=0), the difference is usually negligible.

Q: What's the right approach for fine-tuned models?

Multi-LoRA serving. One base model, many adapter routes. Cost-efficient, operationally simpler than per-adapter replicas.

Q: How do I migrate from one quantization format to another?

Run both in parallel. Compare quality on your eval set. Compare throughput. Migrate when the new format is better on both axes (or one axis with acceptable tradeoff).

Glossary

Continuous batching: dynamic merging of new requests into in-flight batch. Standard since 2022.
Decode: phase 2 of generation. Memory-bandwidth-bound.
EAGLE-2: dominant speculative decoding variant in 2026.
Eviction: removing a sequence's KV from cache. Recompute or swap.
FlashAttention: memory-efficient attention kernels. FA-3 current.
GQA: Grouped-Query Attention. KV memory savings architecture.
Inter-token latency (ITL): time between consecutive output tokens.
KV cache: per-token key/value vectors stored across layers.
LoRA: Low-Rank Adaptation. Small fine-tuning adapters.
Multi-LoRA serving: many LoRAs on one base model.
PagedAttention: KV with fixed-size blocks. vLLM's contribution.
Prefill: phase 1 of generation. Compute-bound.
Prefix caching: deduping KV blocks across shared-prefix requests.
RadixAttention: SGLang's tree-based prefix sharing.
Speculative decoding: drafting K candidates, verifying in one target pass.
Static batching: collecting N requests, running batch to completion. Pre-2022.
TP / PP / EP / DP: tensor / pipeline / expert / data parallelism.
Time to first token (TTFT): latency from request to first output.
vLLM: most popular open-weight serving stack.

References

Foundational papers

PagedAttention / vLLM — Kwon et al., 2023. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv:2309.06180. SOSP 2023. The paging mechanism that became the default KV-cache layout for open serving stacks.
Orca — Yu et al., 2022. "Orca: A Distributed Serving System for Transformer-Based Generative Models." USENIX OSDI 2022. Introduced iteration-level (continuous) batching.
FlashAttention — Dao et al., 2022. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135. The kernel underneath modern attention.
FlashAttention-2 — Dao, 2023. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." arXiv:2307.08691.
Speculative decoding — Leviathan et al., 2022. "Fast Inference from Transformers via Speculative Decoding." arXiv:2211.17192.

Production systems and scheduling

SGLang / RadixAttention — Zheng et al., 2023. "Efficient Execution of Structured Language Model Programs." arXiv:2312.07104. Shared-prefix caching for chat and structured outputs.
DistServe — Zhong et al., 2024. "DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving." arXiv:2401.09670.
Splitwise — Patel et al., 2023. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting." arXiv:2311.18677.
Mooncake — Qin et al., 2024. "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving." arXiv:2407.00079.

Open-source stacks

vLLM — github.com/vllm-project/vllm. Reference implementation of PagedAttention plus continuous batching, chunked prefill, prefix caching, and multi-LoRA.
TensorRT-LLM — github.com/NVIDIA/TensorRT-LLM. NVIDIA's high-performance engine for Hopper/Blackwell.
SGLang — github.com/sgl-project/sglang. RadixAttention runtime plus structured-output frontend.

Background reading

Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Ainslie et al., 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245.

Serving stack feature matrix (2026)

A precise comparison of feature availability across major serving stacks as of mid-2026.

Feature	vLLM 0.7	SGLang 0.4	TRT-LLM 0.13	TGI 2.4	LMDeploy 0.6	llama.cpp
Continuous batching	Yes	Yes	Yes	Yes	Yes	No
PagedAttention	Yes	Yes	Yes (paged_kv)	Yes	Yes	No
Prefix caching	Yes	Yes (Radix)	Yes	Yes	Yes	Limited
Chunked prefill	Yes	Yes	Yes	Yes	Yes	No
Speculative decoding	Yes (EAGLE-2, Medusa)	Yes	Yes (Medusa, ReDrafter)	Yes	Yes	Limited
Multi-LoRA	Yes (Punica)	Yes	Yes (TensorRT-LoRA)	Yes	Yes	No
FP8 KV	Yes (e4m3)	Yes	Yes	Yes	Yes	No
FP4 weights	Yes (B200)	Yes (B200)	Yes (B200)	Limited	Limited	No
Disaggregated prefill	Experimental	Experimental	Yes (NIM)	No	No	No
OpenAI-compatible API	Yes	Yes	Via Triton	Yes	Yes	Yes (server)
Multi-modal (vision)	Yes	Yes	Yes	Yes	Yes	Partial
AMD MI300X	Yes (ROCm 6.2+)	Yes	No	Yes	No	Yes
TPU	Limited (jax-vllm)	No	No	No	No	No
Apple Silicon	No (CPU only)	No	No	No	No	Yes (MLX)
CUDA Graph capture	Yes	Yes	Yes	Yes	Yes	No
Prometheus metrics	Yes	Yes	Yes (Triton)	Yes	Yes	Manual

For most teams in 2026: vLLM is the safe default. SGLang for chat with shared prompts. TensorRT-LLM for max throughput on NVIDIA at scale. llama.cpp for local/edge.

Latency budget breakdown

A request's end-to-end latency decomposes into specific stages. Knowing the rough budget tells you where optimization pays off.

TTFT decomposition (Llama-3 70B, 4K prompt, H100 TP=2)

Stage	Time	Optimization opportunity
Network ingress	1-5 ms	CDN edge, geographic placement
Tokenization	5-15 ms	Cached tokenizers, Rust impl
Queue wait	0-2000 ms	Capacity, admission control
Prefill (4K)	200-400 ms	TP, FP8, FlashAttention-3
First token emit + stream setup	5-10 ms	HTTP/2, SSE buffering
P50 TTFT	~250 ms
P99 TTFT	~2500 ms	Tail dominated by queue and long prefills

ITL decomposition (Llama-3 70B decode)

Stage	Time	Notes
HBM read (weights + KV)	25-30 ms	Bandwidth-bound
Compute (matmul + attention)	3-5 ms	Compute is fast at BS=1
NCCL AllReduce (TP=2)	0.5-1 ms	NVLink-bounded
Python overhead	0.5-2 ms	Killed by CUDA Graphs
P50 ITL	~30 ms	At BS=1
At BS=32	~50 ms	Throughput-batch trade

What's worth optimizing

Tokenization (5-15 ms) matters only at short prompts. Queue wait dominates tail latency under load — capacity buys the most improvement here. Prefill dominates TTFT at long prompts — chunked prefill keeps it bounded. ITL is HBM-bandwidth-bound; the only way to halve it is to upgrade hardware (H200, B200) or quantize KV.

SLO design and queueing math

Production serving SLOs are typically expressed in terms of TTFT and ITL percentiles. Setting them correctly requires understanding the queueing dynamics.

Typical SLO templates

Chat (interactive): P50 TTFT < 500 ms, P99 TTFT < 2000 ms, P50 ITL < 50 ms, P99 ITL < 100 ms.
Code completion: P99 TTFT < 300 ms (users expect near-instant), ITL not user-visible.
Agentic / tool-use: TTFT lax (1000+ ms acceptable), ITL strict (these workloads decode many tokens).
Batch / async: throughput-bound, no per-request latency SLO.

Little's Law for capacity planning

Concurrent requests = throughput × average latency. For 1000 req/min and 5-second average latency, you need 1000/60 × 5 = 83 concurrent slots. Each H100 TP=2 replica handles ~64 slots for Llama 70B — so 2 replicas at minimum, 3 for headroom.

Tail latency under load

P99 latency scales much worse than mean under load. At 70% utilization, P99 is typically 3-5× mean. At 90% utilization, P99 is 10-20× mean. Production SLO targets should size for P99 at peak load, not mean at average load — usually 1.5-2× the naive capacity calculation.

Admission control

When the queue exceeds a threshold, reject new requests (HTTP 429) rather than let them sit until SLO violation. Better to fail fast than to fail late. Most stacks support --max-num-seqs for in-flight cap; add a queue-depth limit at the API gateway.

Production debugging playbook

Common production incidents and their typical resolutions.

"P99 TTFT suddenly spiked to 10 seconds"

(1) Check queue depth — if growing, you're capacity-constrained, scale out. (2) Check for one giant request blocking the batch — chunked prefill should fix. (3) Check GPU utilization — if low, NCCL is hanging or a rank failed. (4) Check tokenizer p99 — long prompts with rare characters can be 100× slower to tokenize.

"Throughput dropped 50% after deploy"

(1) Diff vLLM/SGLang versions — sometimes defaults change. (2) Check CUDA Graph capture — if disabled, dispatch overhead doubles. (3) Check FP8 quantization — sometimes silently downgrades to BF16 on incompatible kernels. (4) Check KV cache size — if smaller, fewer concurrent requests fit.

"Replicas keep OOMing under load"

(1) Lower max_num_batched_tokens. (2) Check for KV cache leak — nvidia-smi should not grow indefinitely. (3) Check FlashAttention version — older versions used more workspace. (4) Reduce gpu_memory_utilization from 0.95 to 0.90.

"Random NCCL timeouts during traffic spike"

(1) Increase NCCL_TIMEOUT to 600s. (2) Check fabric — ibstat and ibcheckerrors. (3) Check for a slow node — per-rank step timing log. (4) Reduce TP if cross-NUMA — co-locate TP ranks on same NUMA node.

"Multi-LoRA quality regressed"

(1) Check adapter compatibility with base model version. (2) Check Punica kernel version — early versions had bugs at high concurrency. (3) Verify adapter rank matches what was trained. (4) Test single-adapter serving — if quality is fine alone but bad multiplexed, it's a kernel issue.

Extended FAQ

Q: How do I size capacity for variable prompt lengths?

Compute weighted average prompt length and weighted average output length from your traffic logs. Use these in capacity calculations. For very variable workloads (some 100-token prompts, some 30K-token prompts), capacity-plan for the long-tail — short prompts are essentially free, long prompts dictate memory.

Q: When should I use TP=2 vs TP=4 vs TP=8 for Llama 70B inference?

TP=2 on H100/H200 fits the model with comfortable KV headroom and minimal NCCL overhead — best for high concurrency. TP=4 doubles per-replica throughput at modest NCCL cost — best for latency-critical workloads. TP=8 within one DGX is overkill for 70B but useful for 405B or for keeping KV cache room for long context. Almost never use TP > 8 — cross-node NVLink isn't there.

Q: How does FP8 KV cache affect prefix caching?

FP8 KV caches are still hashable per-block, so prefix caching works identically to BF16. The only nuance: scale factors per-block must be cached alongside the data; modern stacks handle this transparently. Net effect: 2× KV capacity for free with prefix caching intact.

Q: What's the right `gpu_memory_utilization` for vLLM in production?

0.90 is the sweet spot. 0.95 squeezes maximum KV capacity at the risk of OOM under spike load. 0.85 wastes capacity but is safer for production where occasional huge requests arrive. Never go above 0.95 — there's no headroom for CUDA allocator fragmentation.

Q: How do I deal with cold-start latency for autoscaling?

(1) Pre-warmed pool of standby replicas (cost = idle GPU time). (2) Model preloading — load weights from local NVMe in seconds rather than from S3 in minutes. (3) Cluster autoscaling at the node level, not the pod level — keep nodes warm with placeholder pods. (4) Hybrid hosted+self — burst to hosted API (Bedrock, Vertex) during cold-start window.

Q: What's the realistic throughput on a single 8× H100 node serving Llama 70B?

With vLLM 0.7 + FP8 weights + FP8 KV + chunked prefill + speculative decoding: ~5000-8000 aggregate tokens/sec at P99 ITL < 100 ms. Without speculative decoding: ~3500-5000. Numbers from production deployments, not synthetic benchmarks.

Q: When does speculative decoding hurt instead of help?

Three cases: (1) Very low batch sizes (BS=1) where decode is purely memory-bandwidth bound — speculative adds compute that doesn't translate to time savings. (2) Workloads with high acceptance variance — adversarial prompts (code, math) reject speculation more, raising worst-case latency. (3) When draft model quality is bad — high reject rate means wasted compute on rejected drafts.

Q: How do I cache responses safely at the API layer?

Hash the (model, prompt, sampling_params, tools) tuple — cache the response. Honor temperature > 0 by never caching (results are non-deterministic). For chat with system prompts, cache key includes the system prompt. Cache hit rate of 5-20% is realistic on production traffic; some chat applications see 40%+ with aggressive caching.

Q: What's the deal with `enable_prefix_caching` in vLLM?

Enables block-level prefix sharing across requests. Default off in vLLM 0.6, default on in 0.7+. Benefit: 2-10× throughput for workloads with shared prompts (chat with system prompts, multi-shot agents). Cost: slight memory overhead for the hash table. Always enable unless you have a specific reason not to.

Q: Can I serve a model larger than fits on a single node?

Yes via pipeline parallelism — split layers across nodes. vLLM supports PP since 0.5. Cost: cross-node latency adds 1-3 ms per layer transition. For 80-layer 70B at PP=2, that's 80-240 ms added to TTFT and ITL. Usually not worth it; just use multiple replicas of the same model on each node.

Q: How does serving change for very long context (>128K)?

KV cache dominates memory. Per-token KV at 128K context for Llama 70B GQA-8: ~330 KB. At 128K context that's 42 GB per request — half a node. Capacity drops sharply; concurrency falls to 4-8 per 8-GPU node. Mitigations: aggressive KV quantization (FP8 mandatory, INT4 for some workloads), KV offload to host memory (slow but capacity-rich), architecture choices (sliding window, MoE, hybrid).

Q: What's the right approach for serving 1000+ fine-tunes?

Multi-LoRA with Punica kernels. One base model in HBM, LoRA adapters loaded from disk on demand. Punica's segmented matmul fuses multiple adapters per forward pass. Production deployments serve 200-1000+ adapters concurrently with <10% throughput overhead versus base-model-only.

Q: How do I evaluate serving stack quality regressions?

(1) Pin a reference seed and prompt set; record outputs. (2) After every stack upgrade, re-run; diff outputs. (3) Tolerate small differences (FP8 quantization is non-deterministic) but investigate large differences. (4) Run downstream evals (MMLU, HumanEval) on the served model periodically.

Q: What's `--enforce-eager` in vLLM and when should I use it?

Disables CUDA Graph capture. Slower (5-15% throughput loss) but more flexible — supports variable batch shapes that defeat graph capture. Use only when debugging or for highly variable workloads (continuously-arriving long-context requests). Production: always disable enforce-eager.

Q: How do I serve quantized and unquantized versions of the same model?

Two replicas. There's no efficient way to serve both in one process — they have different weight layouts and kernels. Route traffic at the API gateway based on a header or path. Common: quantized for free tier, unquantized for paid.

Q: What's the realistic latency for tool-using LLMs?

Each tool call breaks the decode loop: model emits tool-call tokens, server stops decoding, dispatches tool, waits for result, resumes decode with tool result in context. Round-trip per tool call: 100 ms (tool execution) + new prefill (50-200 ms depending on context growth) + decode. For agents making 10+ tool calls, total latency is dominated by tools, not LLM. Speculative tool-result decoding (research) tries to predict tool outputs.

Q: How do I handle a model with mismatched tokenizer between SFT and serving?

Don't. Always serve with the exact tokenizer used in training. Tokenizer drift causes weird quality regressions that are hard to diagnose. If you must change tokenizers, retrain or do an explicit token-embedding remap step.

Q: What's the impact of `temperature=0` (greedy) on production serving?

Deterministic outputs (mostly — see below). Better for testing, caching, eval. Slightly different KV cache usage (no sampling buffer). Quality is fine for most tasks but slightly worse on creative writing. Note: even at temperature=0, FP8 quantization can introduce per-batch nondeterminism due to floating-point ordering.

Q: How do I deal with prompt injection attacks?

Two layers. (1) Input filtering — strip known injection patterns, escape user content in system prompts. (2) Output filtering — check generated tokens against a safety classifier. For agentic systems, also: tool-call validation, output structure enforcement (Outlines, LMFE). No single defense is perfect; layer them. See production safety guardrails.

Q: When should I use vLLM vs SGLang for chat?

SGLang for chat with strong prefix sharing (heavy system prompt reuse). vLLM otherwise. Test both with your traffic — SGLang's RadixAttention dominates when chat replays a 2K-token system prompt across thousands of requests; vLLM matches or beats SGLang on workloads without strong prefix structure.

Q: How does multi-modal (vision) inference change capacity planning?

Each image consumes 256-1024 vision tokens in the context. A 4-image conversation may have 4K tokens of vision before any text. KV cache budget needs to account for this. Practical: budget 1024 tokens per image, plan accordingly.

Q: What's the right strategy for serving on AMD MI300X?

vLLM 0.7 with ROCm 6.2+ supports MI300X production-ready. Throughput on 70B BF16 is competitive with H100 TP=2; MI300X's 192 GB HBM lets you serve in TP=1 (cheaper). FP8 support is partial — some kernels still fall back to BF16. AMD is real for inference in 2026; just verify your specific model architecture is supported.

Q: How does INT4 weight quantization compare to FP8?

INT4 (AWQ, GPTQ): 4× weight memory savings vs BF16, ~3-4× decode throughput. Quality: 0.5-1.5 point MMLU drop with good calibration. Production-ready for chat workloads, careful for retrieval/code. FP8: 2× memory savings, 2× decode throughput, near-zero quality drop. Production-default on H100/H200/B200. FP4 on Blackwell: 4× memory savings, 2.5× decode throughput, 0.5-1 point quality drop. Becoming standard mid-2026.

Q: Should I worry about GPU clock instability during inference?

Yes for SLO-sensitive workloads. GPU thermal throttling can cause 10-30% throughput drops mid-traffic. Mitigations: monitor nvidia-smi --query-gpu=clocks.gr,temperature.gpu, alert on clock changes, fix cooling issues quickly.

Q: What's the cheapest hosted alternative when self-hosting is overkill?

For <100M tokens/month, hosted APIs (OpenAI, Anthropic, Bedrock, Vertex) are cheaper than self-hosting. Specifically: at $3/1M tokens for Llama 70B on Together AI versus $4/hr for an H100 serving 5000 tok/s, hosted breaks even at ~17M tokens/hr of sustained traffic.

Q: How do I migrate from OpenAI to a self-hosted Llama?

Three steps. (1) Audit your prompts — many "OpenAI-tuned" prompts work poorly on Llama; rephrase. (2) Pick a Llama variant matching your tasks (Llama 3.3 70B Instruct for chat, Code Llama for code). (3) Run a shadow eval — serve identical queries to both, diff outputs, measure quality regression. Typical: 5-15% quality regression on average, much higher variance per task.

Q: What's the role of CUDA Graphs in serving?

CUDA Graphs capture a sequence of CUDA operations into a replayable graph. For decode steps with fixed shapes (e.g., BS=32, seq_len=1), the graph is replayed each iteration, saving ~5-10 ms per step of dispatch overhead. For Llama 70B at BS=32, that's a 20-30% throughput improvement. Caveats: graphs lock in shape, so dynamic batch sizes defeat capture. Modern stacks (vLLM, TensorRT-LLM) capture multiple graphs at common BS values.

Q: How does serving change for MoE models?

EP within node is common; experts are sharded across TP ranks. Each token routes via all-to-all to its assigned experts. Practical effect: TP=8 for Mixtral 8×22B inference on 8× H100. Compared to dense 70B serving, MoE inference has higher peak FLOPs but more bisection-bandwidth pressure. See mixture-of-experts serving.

Q: What's the impact of HTTP/2 vs HTTP/1.1 for streaming?

HTTP/2 multiplexes streams on one connection, eliminating connection setup per request. For chat workloads with many short interactions, that's 50-200 ms saved per request via connection reuse. SSE over HTTP/2 is the modern default. Always enable.

Q: How do I handle request cancellation?

Client closes the connection. Server should detect (write to closed socket → EPIPE) and stop generating for that request. Most stacks support this; some leak GPU cycles for up to one decode step after cancellation. For pay-per-token billing, refund users for cancelled-mid-decode tokens.

Q: What's the right approach for reasoning models (o1, R1-style)?

Decode budget is much larger — 5K to 50K tokens of "thinking" before visible output. KV cache pressure is significant. Best practice: dedicate capacity per reasoning request, don't multiplex with chat. See reasoning model serving.

Q: When should I use TGI (Text Generation Inference) instead of vLLM?

When you're deep in the Hugging Face ecosystem — TGI integrates tightly with HF Hub, has good ergonomics for model swapping, and supports custom HF model architectures. Performance is competitive with vLLM but typically 10-20% lower at peak throughput. Pick TGI for HF-heavy teams, vLLM for raw throughput.

PagedAttention mechanics deep dive

The internals of PagedAttention as implemented in vLLM, with the engineering details that matter for production.

Block manager

The block manager is the central allocator. It maintains:

A pool of fixed-size memory blocks (typically 16 tokens each, sometimes 8 or 32).
A free list of available blocks.
A per-request mapping from logical sequence positions to physical blocks.
A reference counter for blocks that may be shared (prefix caching).

When a request needs more KV space, the block manager allocates from the free list. When a request completes, its blocks return to the free list. Fragmentation is bounded because all blocks are the same size.

Swap policy

For long sequences or memory pressure, blocks can be swapped to CPU memory:

Swap-out trigger: GPU KV memory below a configured threshold.
Swap-out victim selection: typically least-recently-used sequence, or paused sequences (waiting for tool results).
Swap-in: blocks returned to GPU when the sequence resumes.

vLLM's swap is implemented but rarely used in latency-sensitive deployments; the CPU<->GPU transfer cost is usually larger than the benefit. For throughput-oriented batch deployments, swap can help fit more concurrent sequences.

Copy-on-write

When two sequences share a prefix (prefix caching), they share blocks via reference counting. When one sequence diverges, the diverging block is copied:

Initial state: both sequences point to the same physical block.
Divergence: when sequence A writes a new token, the block is copied; A's pointer updates to the copy.
Sequence B keeps the original.

Copy-on-write is efficient because the copy happens only at the divergence point; most of the shared prefix remains shared.

Block-size trade-offs

The block size (typically 16 tokens) is a trade-off:

Smaller blocks: less internal fragmentation, better memory utilisation.
Larger blocks: less metadata overhead, better attention kernel efficiency.

The vLLM default of 16 is a balance; some deployments tune this per model.

Allocation algorithms

First-fit: allocate the first free block; fastest.
Best-fit: less relevant since all blocks are equal size.
Power-of-two grouping: in some advanced variants.

For most deployments, the first-fit default is fine.

Continuous batching scheduler in detail

The scheduling logic that determines which requests run on which step.

Greedy / FCFS

First-come, first-served. Simple, fair, no priority handling.

Priority

Some stacks support per-request priority. Higher-priority requests get preferred scheduling. Useful for differentiating chat vs background tasks.

Iteration-level scheduling

The defining feature of continuous batching. Each iteration (one decode step), the scheduler:

Checks for new arrivals; admits if budget allows.
Removes completed requests.
Computes the new batch.
Issues forward pass.

The "batch" can change every iteration. This is the productivity gain vs static batching.

Admission control

Decisions about which new requests to admit:

KV budget: don't admit if KV would exceed available memory.
Latency budget: in latency-sensitive deployments, don't admit if it would push existing requests beyond SLO.
Token budget: limit per-step total tokens to keep forward pass cost predictable.

Backpressure

When the system is at capacity, what happens to new requests:

Queue: wait in line; risk of long tail latency.
Reject: return 429 / Too Many Requests; client can retry.
Shed: drop low-priority requests.

Production patterns balance these based on SLO and elasticity.

Preemption

For very long requests blocking shorter ones:

Preempt the long request (save state); resume later.
Cost: state-save + state-restore overhead.

Used in some deployments; less common in mainstream serving.

Prefix caching mechanics

Detailed mechanics of prefix caching across stacks.

vLLM v0.6 prefix caching

Implemented as block-level reference counting. When a new request arrives, vLLM checks if its prefix matches any cached blocks. Matched blocks are reused.

Limitations of v0.6:

LRU eviction may discard hot prefixes under pressure.
Hash-collision corner cases require care.

vLLM v1 prefix caching

Improved cache management with better eviction policy and more robust hashing. Throughput improvements in production workloads with prefix-heavy traffic (system prompts, few-shot exemplars).

SGLang RadixAttention

Radix-tree-based representation of cached prefixes. The radix tree natively expresses shared and divergent prefixes.

Advantages:

Efficient lookup.
Natural handling of branching prefixes (different completions of the same prompt).
Strong fit for agentic workloads with tree-shaped prompts.

TensorRT-LLM prefix caching

Implemented through the engine's KV cache management; configurable per engine build. Performance is competitive with vLLM/SGLang.

Cache hit rate metrics

Track:

Cache hit rate: percent of requests with prefix matches.
Cache savings: prefill tokens saved by cache hits.
Cache size: how much memory the cache uses.

For prefix-heavy workloads (chat with system prompts, few-shot scenarios), cache hit rates can exceed 70%, saving substantial prefill cost.

FlashAttention-3 paged kernel

FlashAttention-3 (released 2024) is the production attention kernel for H100/H200 GPUs. The paged variant integrates with PagedAttention:

Supports non-contiguous KV layouts (i.e., paged KV).
Optimised for FP8 KV with quantised storage.
Uses the warp-specialisation and producer-consumer pattern of Hopper GPUs.

Benefits over FlashAttention-2:

~1.5–2x faster on Hopper.
Better FP8 support.
Cleaner integration with paged-attention.

For Blackwell (B100/B200), the corresponding kernel evolution is FlashAttention-4 or successor; details are evolving through 2025–2026.

For background on attention kernels see kv cache.

Per-feature matrix

The detailed feature matrix across major serving stacks as of mid-2026.

Feature	vLLM	SGLang	TRT-LLM	TGI	LMDeploy	llama.cpp
Paged KV	Yes	Yes (Radix)	Yes	Yes	Yes	Limited
Prefix cache	Yes (v0.6+)	Yes (Radix)	Yes	Yes	Yes	Limited
Continuous batching	Yes	Yes	Yes	Yes	Yes	Limited
Speculative decoding	Yes (multi-variant)	Yes	Yes	Limited	Yes (Medusa)	Yes (draft)
LoRA serving	Multi-LoRA	Multi-LoRA	Multi-LoRA	LoRA	LoRA	LoRA
Structured outputs	Native (xgrammar)	Native	Yes	Limited	Limited	Limited
Beam search	Limited (deprecated)	Limited	Yes	Yes	Yes	Yes
Async tokenisation	Yes	Yes	Yes	Yes	Limited	Limited
KV cache offload (CPU)	Yes	Yes	Yes	Limited	Limited	Yes
Multi-LoRA concurrent	Yes	Yes	Yes	Limited	Yes	No
FP8 KV	Yes	Yes	Yes	Limited	Yes	Limited
FP8 weights	Yes	Yes	Yes	Limited	Yes	Yes
INT8 KV	Yes	Yes	Yes	Limited	Yes	Yes
KIVI (INT2) KV	Research	Research	No	No	No	No
MoE serving	Yes	Yes	Yes	Limited	Yes	Limited
Vision (VLM)	Yes	Yes	Yes	Limited	Yes	Limited
Chunked prefill	Yes	Yes	Yes	Limited	Yes	No
Disaggregated PD	Adding	Adding	Yes	No	No	No
Kubernetes operator	Adding	Community	NIM	Community	Community	No

Matrix entries change rapidly; treat as approximate for mid-2026.

KV quantisation in serving

KV cache quantisation is one of the highest-leverage memory optimisations.

FP8 KV

Reduces KV cache memory by 2x (from BF16/FP16 to FP8).
Negligible accuracy impact on most models.
Supported on H100/H200 hardware with FP8 hardware path.
Widely deployed in 2026.

INT8 KV

Similar memory reduction; slightly more accuracy impact.
Useful on hardware without FP8 (A100).
Less common than FP8 on current hardware.

KIVI (INT2)

4x memory reduction.
More accuracy impact; requires careful calibration.
Research-stage in mainstream stacks.

INT4 KV

4x memory reduction.
Accuracy impact more pronounced; per-model qualification needed.
Some specialised stacks support.

Trade-offs

Lower-precision KV reduces memory but can:

Lower acceptance rate in speculative decoding.
Reduce quality on hard prompts.
Require per-model calibration.

For most production deployments, FP8 KV is the safe sweet spot.

MoE serving in detail

Mixture-of-experts models (Mixtral 8x7B, 8x22B, DeepSeek-V3, GPT-4-class) have specific serving requirements.

Expert parallelism (EP)

Experts sharded across TP/EP ranks. Each token routes to its assigned experts via all-to-all communication.

Bisection bandwidth pressure

MoE serving creates more all-to-all traffic than dense serving. Inter-GPU bandwidth (NVLink, NVSwitch) matters more.

Routing patterns

Token-to-expert routing creates imbalance: some experts hot, some cold. Production schedulers may consider expert utilisation.

KV cache for MoE

KV cache is per-token (not per-expert), so KV memory math is similar to dense. The compute cost is what differs.

Quantisation for MoE

FP8 weights for MoE experts; FP8 KV. Both supported in vLLM, SGLang, TRT-LLM.

Speculative decoding for MoE

Dense draft is simplest. See speculative decoding for the full picture.

Vision-language model serving

Vision-language models (LLaVA family, Llama 3.2 Vision, GPT-4V, Claude Vision, Gemini multimodal) have specific serving considerations.

Image encoding

Image is processed through a vision encoder (ViT or similar) producing visual tokens. These are then mixed with text tokens in the language model's context.

KV cache implications

Visual tokens consume KV cache like text tokens. Longer visual context (high-res images, multiple images) = more KV.

Prefill cost

Vision encoder + visual-token prefill is a larger prefill than pure text. Throughput-per-image is lower than throughput-per-text-token.

Batching

Mixed text and vision requests in a batch require careful scheduling. Some stacks batch vision and text separately.

Memory budget

Per-image visual-token count (often hundreds to thousands of tokens per image) makes large multi-image requests memory-intensive.

Supported stacks

vLLM, SGLang, TRT-LLM all support major VLMs by mid-2026. Llama.cpp has limited VLM support.

Throughput vs latency math

The fundamental trade-off and how to model it.

Throughput formula

Throughput = batch_size × tokens_per_second / sequence_length

Higher batch size = higher throughput.

Latency formula

Per-request latency = sequence_length × time_per_token + queueing_time

Higher batch size = higher time_per_token (more contention) but lower queueing time.

The sweet spot

For a given workload (request rate, sequence length distribution, SLO), there's an optimal batch size that balances throughput and latency.

Modelling

Build a queueing model:

Arrival rate (requests/second).
Service rate (requests/second, depends on batch size).
SLO target.

Use M/M/1 or M/M/c approximations as a starting point; refine with actual measurements.

Production patterns

Conservative SLO → small batch, low throughput, low latency.
Aggressive SLO → larger batch, higher throughput, higher latency tolerance.
Mixed → multiple deployments at different points on the curve, routing per workload.

SLO design across percentiles

The SLOs that matter for LLM serving.

TTFT (Time To First Token)

P50, P95, P99. Dominated by prefill cost. Targets vary by use case: <500ms for chat; <100ms for completion features; multi-second OK for batch.

ITL (Inter-Token Latency)

P50, P95, P99 between consecutive tokens. Dominated by decode cost. Targets vary: <30ms for fluent chat; <10ms for premium experiences.

TPOT (Time Per Output Token)

Aggregate measure of decode speed.

TTLT (Time To Last Token)

End-to-end latency for full response.

Throughput per GPU

Operational metric for capacity planning.

Cost per million tokens

Business metric; ties to capacity and SLO targets.

Trade-offs

Lower-latency SLOs require smaller batch sizes and more GPUs for the same throughput. Higher SLOs allow larger batches and better economics.

Failure mode taxonomy

The ways LLM serving fails in production, organised.

Capacity failures

Out-of-memory on GPU.
Queue depth exceeded.
Rate limit triggered.

Quality failures

Wrong output (hallucination).
Unsafe output (jailbreak success).
Inappropriate refusal.

Performance failures

Tail latency spike.
TTFT regression.
Throughput drop.

Infrastructure failures

GPU hardware failure.
Network partition.
Driver / kernel issue.

Software failures

Stack version mismatch.
Model load failure.
Tokeniser bug.

Operational failures

Misconfiguration.
Deployment regression.
Bad model rollout.

Mitigation patterns

Health checks and circuit breakers.
Canary deployments.
Multi-region failover.
Rate limiting at edge.
Observability with alerting.

Observability deep dive

What to instrument for production LLM serving.

Prometheus metrics

Request rate.
Tokens-per-second (input and output).
Active requests.
Batch size distribution.
KV cache utilisation.
GPU utilisation.
HBM bandwidth utilisation.
Per-request latency (TTFT, ITL, TTLT).
Cache hit rate.
Error rate.

Distributed tracing

OpenTelemetry traces from API edge through serving stack:

Request arrival.
Tokenisation.
Admission to batch.
Forward passes (per step).
Streaming response.

Logging

Per-request logs with sampled tokens (for debugging).
Error logs with stack traces.
Slow-query logs (above latency threshold).

Dashboards

Real-time throughput and latency.
Capacity utilisation.
Error rates.
Per-model breakdowns.

Alerting

SLO breaches.
Error rate spikes.
Capacity thresholds.
Anomalous behaviour.

Deployment patterns deep dive

Production deployment architectures.

Kubernetes + custom operator

The default for many organisations. Pros: portable, mature ops tooling. Cons: GPU operator complexity, networking nuances.

KServe

Kubernetes-native model serving. Inference Service abstraction. Good for multi-model serving.

Ray Serve

Python-native serving framework. Good for teams already on Ray for training.

BentoML

Python-friendly framework with growing LLM support.

NVIDIA Triton

NVIDIA's serving framework. Supports multiple backends including TRT-LLM. Common in NVIDIA-heavy enterprises.

NVIDIA NIM

Higher-level NVIDIA-hosted inference. Faster path to deployment; less control.

Cloud-native managed

Azure OpenAI Service, AWS Bedrock, Google Vertex AI. Managed service. Less control, faster deployment.

Self-hosted vLLM

Direct vLLM deployment. Most control; most operational overhead.

Choosing a pattern

Speed to deploy → managed service.
Cost control → self-hosted vLLM.
Multi-model → KServe or Triton.
Python-native team → Ray Serve or BentoML.
Established Kubernetes shop → custom operator.

Cost arithmetic per stack

Approximate cost-per-million-tokens calculations.

Self-host vLLM on rented GPUs

8x H100: ~$25–$30/hour on major clouds.
Llama 70B throughput: ~5000 tokens/sec aggregate.
Cost-per-million-tokens: ~$1.50–$2.

Cloud-native API

OpenAI / Anthropic API: $2–$15 per million tokens, varies by model.
Substantial margin over self-host raw cost; includes provider's overhead.

Specialised inference provider

Together, Fireworks, Anyscale: $0.20–$1.50 per million for open-weight models.
Substantial discount vs frontier API.

Cost comparison

Self-host raw < specialised provider < API < frontier API. The trade-off is operational complexity.

For full cost economics see AI inference cost economics.

Benchmarks per stack

Per-GPU throughput benchmarks for common models (mid-2026, approximate).

Llama 70B (FP8, 8x H100, TP=8)

vLLM: ~4000–5000 tokens/sec aggregate.
SGLang: similar.
TRT-LLM: ~5000–6000 tokens/sec (with engine tuning).
TGI: ~3000–4000 tokens/sec.
LMDeploy: ~4000–5000 tokens/sec.

Mixtral 8x22B (FP8, 8x H100, TP=8, EP=8)

vLLM: ~3000–4000 tokens/sec.
SGLang: similar.
TRT-LLM: ~4000–5000 tokens/sec.

DeepSeek-V3 (full FP8, multi-node)

vLLM: production deployments achieving several thousand tokens/sec per node.
TRT-LLM: optimised builds achieving higher.

Numbers caveat

Benchmarks vary by sequence length, batch composition, prefix cache hit rate, and many tuning parameters. Treat published numbers as rough order; benchmark on your workload.

When to roll your own serving stack

Most organisations should not roll their own. Reasons to consider:

You're a frontier lab with capability beyond mainstream stacks.
You have unique requirements (specific hardware, novel architectures).
The mainstream stacks have a structural mismatch with your workload.

Reasons not to:

Maintenance burden is enormous.
The mainstream stacks have benefitted from years of optimisation.
Your time is better spent on application differentiation.

For most teams, contributing to vLLM or SGLang is better than starting from scratch.

Future direction: vLLM v1 and SGLang RouteLLM

The architectural evolution of the major stacks.

vLLM v1

Cleaner architecture; better dynamic batching.
First-class tree decoding for speculative.
Improved prefix caching.
Better disaggregated-serving support.
Production hardening through 2025–2026.

SGLang RouteLLM

Multi-model routing within one serving stack.
Cheap-model-first with escalation to expensive models.
Useful for cost optimisation on workloads with variable difficulty.

TRT-LLM TensorRT-Engine vs PyTorch path

TRT-LLM's engine-based path remains the highest-performance for NVIDIA hardware.
The PyTorch path adds flexibility at cost of some performance.
Convergence through 2026.

Multi-stack standardisation

OpenAI API compatibility is the de-facto standard.
Many stacks support the same wire format.
Reduces lock-in.

Cross-references

The serving stack intersects with many other parts of the AI infrastructure:

KV cache explained — the data structure serving optimises.
Speculative decoding — the headline decode-time optimisation.
Disaggregated inference — separating prefill and decode pools.
NCCL guide — the communication library underpinning multi-GPU serving.
NVIDIA datacenter GPUs — the hardware tier.
AI training networking — networking patterns shared with serving.
AI inference cost economics — the cost model.
Mixed precision training — precision concepts that carry into serving.
Verifiable inference — attesting to serving outputs.
Production AI safety guardrails — what runs adjacent to the serving stack.

Extra FAQ for serving in 2026

What's the dominant serving stack in mid-2026? vLLM remains the most-deployed open-source stack. SGLang and TRT-LLM are widely used in performance-sensitive deployments. The mainstream open-source landscape is healthy.

Is vLLM v1 production-ready? By mid-2026, v1 is rolling out across deployments. v0.6 remains stable. The migration is non-trivial but the improvements are real.

Should I use a managed service or self-host? Depends on cost, control, and operational maturity. For most organisations starting out, managed services are faster to deploy. For cost-sensitive at-scale workloads, self-hosting pays off.

What's the biggest mistake in LLM serving deployment? Not using continuous batching. Static batching is often the default in naive deployments; it leaves 5–10x throughput on the table.

How do I scale beyond a single node? Tensor parallelism within a node; pipeline parallelism across nodes. Disaggregated prefill/decode for separation. See the multi-GPU section above and disaggregated inference.

Is prefix caching always beneficial? For workloads with shared prefixes (system prompts, few-shot), yes. For workloads without prefix sharing, the cache management overhead may not pay off. Measure cache hit rate.

Should I enable speculative decoding by default? For chat and decode-bound workloads, yes. For batch-only throughput-optimised workloads, the marginal benefit may be smaller.

How do I handle reasoning models in serving? Higher KV pressure; longer decode budget. Dedicated capacity for reasoning workloads; don't multiplex aggressively with chat.

What's the right size for a serving cluster? Sized to peak load with headroom. Autoscaling handles diurnal variation. Reserve capacity for failure modes.

How do I monitor serving health? Prometheus metrics, distributed tracing, structured logs, dashboards, alerts. Build the observability before you need it.

What's the role of CUDA Graphs? Reduces dispatch overhead for fixed-shape decode steps. 5–10ms saved per step adds up at scale.

Should I use FP8 throughout? Generally yes on H100/H200. Weights in FP8 + KV in FP8 + activations as needed. Quality impact is small; throughput and memory wins are large.

How do I handle bursty traffic? Autoscale horizontally; rate-limit at edge; queue with bounded depth. For very bursty, accept some queueing latency.

What's the right approach to long-context serving? Larger KV cache budget; possibly larger blocks; prefix caching for repeated long prompts. For very long context, consider RAG instead of pure long-context.

Does multi-LoRA serving work? Yes, in major stacks. Multiple LoRA adapters concurrently with shared base weights. Memory cost is per-LoRA; overhead is small.

What about structured output serving? xgrammar, outlines, lm-format-enforcer integrated into mainstream stacks. Constrained decoding works with continuous batching.

How do I think about tail latency? Measure P95/P99; design for tail by leaving capacity headroom and avoiding head-of-line blocking. Tail latency is what users feel.

Is gRPC vs HTTP/2 vs WebSocket a meaningful choice? For internal use, gRPC. For browser clients, SSE over HTTP/2 is standard. WebSockets for bidirectional streaming.

How do I handle model updates? Canary deploy new model. Roll out gradually. Monitor metrics. Rollback if needed. Don't replace all serving capacity at once.

What's the deployment pattern for hybrid (cloud + on-prem)? Common in regulated industries. Cloud for elastic capacity; on-prem for sensitive workloads. Routing layer at the edge.

How do I right-size GPUs for my workload? Start with workload profile (concurrent users, request rate, sequence length distribution). Build queueing model. Provision for peak with headroom.

What's the future of inference serving? Continued consolidation of best-practice features into all major stacks. More disaggregated patterns. More fine-grained orchestration (per-token batching variants). Confidential computing for high-stakes serving.

Production case studies (2026)

Anonymised patterns from production deployments.

Frontier lab inference fleet

A frontier lab operates a fleet of thousands of GPUs running custom variants of vLLM and TRT-LLM. Key features in use: paged KV, prefix caching, speculative decoding (custom variants), FP8 throughout, disaggregated prefill/decode, multi-region routing. The cost economics are unpublished but rough order: substantial fraction of total operating expense.

Specialised inference provider (Together, Fireworks, Anyscale)

Open-weight model hosting. Stack typically vLLM with custom optimisations. Multi-model serving with autoscaling. Pricing competitive with frontier APIs at significant discount.

Mid-size SaaS company hosting Llama

vLLM on rented GPUs. Llama 70B FP8 at TP=8 on 8x H100. Throughput ~5000 tokens/sec aggregate. Cost-per-million-tokens ~$2 raw GPU cost. Engineering team of 2–3 manages the deployment.

Enterprise self-host

Mid-size enterprise running vLLM on-prem for data sensitivity. Llama 70B + Mistral Large + DeepSeek-V3 across multiple deployments. Engineering team 4–6. Cost driven by hardware capex amortisation.

Edge / on-device

Apple Intelligence-style on-device AI. Small models (3B–8B) on-device; larger models in private cloud. Combined architecture with strong privacy properties.

Startup with cloud-managed AI

OpenAI or Anthropic API. No serving infrastructure; pay per token. Faster product velocity; higher per-token cost.

Trade-offs summary

Self-host: low marginal cost, high fixed engineering cost. Managed API: high marginal cost, low engineering cost. Specialised provider: middle ground.

The right choice depends on usage volume, technical capability, and strategic considerations.

Disaggregated prefill/decode in production

Disaggregated serving (separate GPU pools for prefill and decode) is becoming standard for high-scale production.

Why disaggregate

Prefill and decode have different compute/memory profiles.
Disaggregating allows independent scaling of each.
Different hardware can be optimal for each (memory-bandwidth GPUs for decode; compute-rich for prefill).

Architecture

Prefill pool: receives full prompt; produces initial KV cache; emits to decode pool.
KV cache transfer: across network (NVLink within rack; InfiniBand or RoCE across racks).
Decode pool: receives KV; runs autoregressive decode; streams tokens to client.

Trade-offs

Extra network bandwidth for KV transfer.
More operational complexity.
Better total throughput at scale.

Stack support

TRT-LLM has mature disaggregated support.
vLLM and SGLang are adding through 2025–2026.
Custom internal stacks at frontier labs.

For the deeper picture see disaggregated inference.

Long-context serving

Serving models with very large context windows (>128k tokens) has specific challenges.

Memory pressure

KV cache for long context can be enormous. A 200k-token context for Llama 70B at BF16 KV is roughly 80 GB; FP8 reduces to 40 GB. Manage carefully.

Prefill cost

Prefilling 200k tokens is computationally expensive. Throughput is bound by prefill on long-context workloads.

Chunked prefill

Split prefill into chunks; mix with decode in the same batch. Improves throughput by keeping decode active during long prefills.

Prefix caching

Long shared prefixes benefit massively from prefix caching. Many long-context workloads have repeated long prefixes (documents, codebases) that cache extremely well.

Practical patterns

For RAG workloads, keep context bounded; rely on retrieval rather than pure long context.
For document-analysis workloads, leverage prefix caching aggressively.
For long-conversation workloads, persistent KV across turns reduces prefill cost on each turn.

Reasoning model serving

Reasoning models (o-series, R1, Claude extended thinking, Gemini Deep Think) have specific serving characteristics.

Long decode budget

Thinking tokens can be thousands per query. Decode-bound by definition.

KV pressure

Long decodes accumulate KV. Memory management is critical.

Throughput math

The "thinking" portion is amortised across the final answer. Per-final-answer-token throughput appears lower than non-reasoning models.

Speculative decoding interaction

Reasoning workloads typically have higher acceptance rates on thinking tokens; speculative decoding has larger absolute benefit on reasoning workloads. See speculative decoding.

Capacity planning

Capacity per reasoning request is roughly 5–10x a chat request. Plan accordingly.

Routing patterns

Separate pool for reasoning workloads; don't multiplex with chat. Reasoning requests have different SLO profile (longer total time, but per-token latency similar).

Multi-model serving

For platforms serving multiple models, additional considerations.

Cold start

Loading a model takes minutes (large models load from disk; weights to GPU memory). Cold starts disrupt SLO.

Hot pool

Keep N most-used models loaded; evict less-used. Trade-off: memory for hot pool vs cold-start cost.

Routing

Request → model classifier → appropriate serving pool. Classifier may be heuristic or model-based.

Cost optimisation

Cheap models for easy queries; expensive models for hard. RouteLLM-style escalation.

Operational complexity

Multi-model is operationally more complex than single-model. Worth it for platforms; not for single-product deployments.

Streaming patterns

Token streaming to clients.

SSE (Server-Sent Events)

The most common pattern for web clients. HTTP/2 + SSE works well. One-way (server to client).

WebSockets

Bidirectional. Useful for chat with interruption (user cancels mid-stream).

gRPC streaming

For internal microservices. Strong typing; efficient binary protocol.

Token batching for streaming

Send tokens in small batches (e.g., every 50ms) rather than every single token. Reduces overhead; user-visible latency unchanged.

Backpressure

Slow client = slow generation? Implementation choice. Most stacks decouple generation from streaming so a slow client doesn't slow others.

Structured output and tool use serving

Constrained generation in production.

Schema enforcement

JSON schema, regex, context-free grammars. Modern stacks integrate xgrammar, outlines, lm-format-enforcer.

Performance impact

Constraints add some overhead (logits masking). Modern implementations are efficient (typically <5% throughput cost).

Function call patterns

Tool/function calling is a structured output. The model produces a function name and arguments in a known schema.

Validation

Server-side validation of outputs. Reject and retry malformed.

Speculative decoding interaction

Constraints reduce draft acceptance somewhat. Modern stacks support both together.

Safety and guardrail integration

Production serving stacks often include safety filters in the request path.

Input filtering

Pre-LLM safety checks: prompt injection detection, content classification.

Output filtering

Post-LLM checks: safety classification, factuality scoring, PII detection.

Integration patterns

In-process: filters run in same process as serving stack.
Sidecar: separate filter service.
Edge: filters at API gateway.

For the deeper guardrail picture see production AI safety guardrails.

Performance considerations

Filtering adds latency. Optimisations: parallel with LLM; cached for repeated patterns; selective per workload.

Hardware choice considerations for serving

The GPU/accelerator landscape for serving in mid-2026.

NVIDIA H100 / H200

The mainstream choice. H100 (80GB HBM3) and H200 (141GB HBM3e) are widely available. H200's larger memory helps long-context and large models.

NVIDIA Blackwell (B100/B200/GB200)

Rolling out through 2024–2026. Significant compute and memory improvements. The leading edge for new deployments.

AMD MI300X / MI325X

Competitive at large memory (192GB+ HBM3e). Software stack (ROCm) maturing. vLLM and SGLang have ROCm support.

Intel Gaudi 3

Strong price/performance for some workloads. Smaller ecosystem.

AWS Trainium / Inferentia

Tightly integrated with AWS Bedrock. Cost-competitive for specific models.

Google TPU

Used internally and via Google Cloud. Less common for self-host scenarios.

Apple Silicon

For small models and edge. M-series Macs run 7B–70B models with quantisation.

Choosing

Mainstream: NVIDIA H100/H200.
Cost-sensitive at scale: AMD or specialised cloud chips.
Edge: Apple Silicon or Jetson.
Cloud-native: cloud-provider-specific chips with their managed services.

For the deeper hardware picture see NVIDIA datacenter GPUs.

The role of compilers and kernels

Custom CUDA/HIP kernels and compilation make a substantial throughput difference.

FlashAttention family

FlashAttention-2 and FlashAttention-3 are the production attention kernels. Paged variants integrate with PagedAttention.

Triton kernels

Many serving stacks have Triton-implemented kernels for specific operations. Easier to develop than CUDA; competitive performance.

CUDA Graphs

Replay sequences of CUDA operations as a single graph. Reduces dispatch overhead. Mainstream stacks use CUDA Graphs for decode steps.

Custom kernels for MoE

MoE all-to-all and expert dispatch benefit from custom kernels (DeepSeek's DeepEP, NVIDIA cutlass kernels).

TensorRT-LLM engine builds

TRT-LLM compiles models into TensorRT engines for maximum NVIDIA performance. Build process takes time but produces optimised inference paths.

Compilation in vLLM

torch.compile integration provides some compilation benefits.

Tuning hot paths

For specific deployments, kernel-level tuning can recover 10–30% throughput. Worth it at large scale; overkill at small scale.

Operational maturity checklist

For teams running production LLM serving, an operational checklist.

Deployment

Reproducible deployment via Terraform / Helm / similar.
Versioned model artifacts.
Canary deployment pattern.
Rollback procedure tested.

Observability

Prometheus metrics exported.
Distributed tracing instrumented.
Structured logs.
Dashboards for key metrics.
Alerting on SLO breaches.

Capacity management

Capacity model documented.
Autoscaling configured.
Headroom for spike absorption.
Cost tracking by workload.

Incident response

Runbook for common failures.
On-call rotation.
Postmortem culture.
Blast-radius limits via rate limiting.

Quality

Quality regression tests on model updates.
User feedback mechanisms.
Hallucination monitoring (for relevant workloads).
Safety filter performance.

Cost optimisation

Periodic right-sizing.
Optimisations enabled (paging, prefix cache, speculative, FP8).
Multi-tenant utilisation tracked.
Provider/hardware refresh cycle planned.

Final 2027 outlook for serving

Probable directions through 2027:

vLLM v1 becomes the default; legacy v0.x deprecated.
Disaggregated serving becomes standard in high-scale deployments.
FP4/INT4 quantisation enters production for KV and weights on Blackwell.
Confidential computing inference reaches production-grade quality.
Multi-model routing within single deployments becomes common.
Speculative decoding becomes universal default.
Edge serving capability expands with smaller, more capable models.

Table of contents

Key takeaways

Mental model: LLM serving in one minute

What "serving" actually means

The two phases: prefill and decode

Chunked prefill

Disaggregated prefill and decode

How the disaggregation works mechanically

When disaggregation pays off

Stack support in 2026

Continuous batching: the headline win

Static batching: the baseline

Continuous batching: the fix

How much it actually buys

Configuring continuous batching

How continuous batching interacts with prefill

Scheduler internals

Continuous batching's memory management

PagedAttention and the KV cache layer

KV cache lifecycle in a serving stack

Block table data structure

Prefix caching and RadixAttention

Where prefix caching pays

Where it doesn't

vLLM's block-level vs SGLang's RadixAttention

Things that break prefix caching invisibly

Cross-replica prefix caching

Speculative decoding

How it works

Variants

When spec-decode wins

When it doesn't

Configuring spec-decode (vLLM)

Quantization at serving time

Stack support matrix

Choosing a format

Multi-LoRA serving

How it works

Stack support

Performance and operational implications

Configuring on vLLM

When multi-LoRA isn't right

How multi-LoRA actually works under the hood

LoRA rank tradeoffs

LoRA adapter loading strategies

Scheduling, admission control, and priority

vLLM's default

Where simple scheduling falls down

Beyond FCFS

Admission control

Preemption

Multi-GPU: TP, PP, EP, DP combinations

Tensor parallelism (TP)

Pipeline parallelism (PP)

Expert parallelism (EP)

Data parallelism (DP)

Combining strategies

Configuring multi-GPU

The major stacks compared

vLLM

SGLang

TensorRT-LLM

TGI (Hugging Face)

LMDeploy

llama.cpp

Decision matrix

Comparative benchmarks (mid-2026)

Stack version stability

vLLM internals: how it actually works

vLLM's continuous-batching internals

Async multi-step scheduling

SGLang internals

TensorRT-LLM internals

A typical production deployment architecture

Latency engineering: prefill, decode, tails

The metrics that matter

Reducing TTFT

Reducing ITL

Managing tail latency

SLO budgets for common applications