Distributed LLM Training: The Complete Guide

A frontier model in 2026 is trained on 10,000+ GPUs for months. None of those GPUs hold the entire model. None of them hold the entire dataset. Coordinating them — splitting the model, splitting the data, keeping them in sync, recovering from inevitable failures — is what "distributed training" actually means.

The vocabulary is dense: DP, TP, PP, EP, SP, FSDP, ZeRO-1/2/3. People use these terms differently across labs and codebases. This guide nails down what each actually splits, when each pays off, and how to reason about combining them.

Table of contents

1. Key takeaways
2. Mental model: distributed training in one minute
3. The fundamental problem
4. Data Parallelism (DP)
5. Tensor Parallelism (TP)
6. Pipeline Parallelism (PP)
7. Expert Parallelism (EP)
8. Sequence and Context Parallelism (SP/CP)
9. Fully Sharded Data Parallel (FSDP) and ZeRO
10. 3D, 4D, 5D parallelism: combining strategies
11. Communication patterns and topology
12. Mixed precision training
13. Gradient accumulation and effective batch size
14. Checkpointing and resharding
15. Fault tolerance at scale
16. The major frameworks
17. Capacity planning a training run
18. Cost economics
19. Failure modes
20. Memory math worked end-to-end
21. Communication-computation overlap
22. Picking parallelism for your model size
23. Cross-DC and federated training
24. Training reproducibility
25. Frontier lab training playbooks
26. Megatron-LM TP, SP, and CP in detail
27. DeepSpeed ZeRO stages 1, 2, 3, Infinity
28. FSDP1 vs FSDP2: the PyTorch 2.6 rewrite
29. Pipeline schedules: GPipe, 1F1B, interleaved 1F1B, Zero Bubble
30. Memory math worked example: Llama 70B
31. DiLoCo for cross-DC training
32. Reference designs at 100, 1000, 10000 GPU scale
33. CPU offload and SWAP: when memory really runs out
34. LocalSGD and async data-parallel variants
35. Frontier training: 2026 case studies
36. Activation checkpointing in detail
37. Cluster utilization metrics: MFU, HFU, MBU
38. The bottom line
39. FAQ
40. Extended FAQ
41. Glossary
42. References

---

Key takeaways

Five orthogonal axes split a training job across GPUs:

1. Data parallelism (DP) — replicate the model, split the batch. Cheap, communication-light, requires fitting the model on one GPU. Standard.
2. Tensor parallelism (TP) — split each weight matrix across GPUs. Required when one GPU can't hold the model. NVLink-bound.
3. Pipeline parallelism (PP) — split the model by layer across GPUs. Bubbles cost throughput; required for multi-node giant models.
4. Expert parallelism (EP) — for MoE only. Distribute experts across GPUs.
5. Sequence/context parallelism (SP/CP) — split the sequence dimension. Needed for very-long-context training.

Plus the activation- and optimizer-state sharding family:

• FSDP / ZeRO — shard model parameters, gradients, and optimizer state across DP replicas. Saves memory at the cost of extra communication.

Modern frontier-scale training combines 3–5 of these axes simultaneously. Llama-3 405B was trained with TP=8 × CP=2 × PP=16 × DP=≥64. The art is choosing the combination that maximizes hardware utilization given your model size, sequence length, and interconnect.

The practical takeaway: you don't pick one strategy. You pick a combination, and the combination is dictated by what your hardware lets you do.

---

Mental model: distributed training in one minute

The named problem is the memory wall: a single GPU cannot hold a frontier model. A 70B model in BF16 is 140 GB of parameters before you add gradients, optimizer states, and activations — call it 1.1 TB end-to-end. An H100 has 80 GB. The model literally does not fit, and shrinking precision or rematerializing activations only buys one generation at a time. Every parallelism strategy is a different answer to "what do we slice, and what do we replicate?"

Conceptually, think of training state as four dimensions: the batch, the layer stack, the weight matrix inside a layer, and the sequence. Each parallelism axis splits exactly one of those dimensions and replicates the rest. Data parallelism splits the batch. Pipeline parallelism splits the layer stack. Tensor parallelism splits each matrix. Sequence/context parallelism splits the tokens. FSDP/ZeRO is the special case that splits parameters themselves across data-parallel replicas, materializing them on demand. Real frontier runs compose 3-5 of these axes at once — a 3D or 4D mesh of GPUs where each axis pays a different communication cost.

Strategy	What it splits	What it replicates	Comms per step	When it pays off
DP	Batch	Full model on every GPU	All-reduce gradients (~params)	Model fits on 1 GPU
TP	Each weight matrix	Layer order, batch	All-reduce activations per layer	NVLink-local, model too big for 1 GPU
PP	Layer stack	Each layer on its stage	Activations between stages	Many layers, slow interconnect
FSDP / ZeRO-3	Params + grads + optim	Compute graph	All-gather params + reduce-scatter grads	Memory-constrained DP
EP (MoE)	Experts	Routing/non-expert layers	All-to-all of tokens	Sparse MoE models
SP / CP	Sequence dim	Weights	Ring all-gather of KV	Long-context training

Conceptually:

# DP: one mesh axis, replicate model, split batch
mesh = make_mesh(dp=64)
# 3D: stack axes, model lives on a 3D grid of GPUs
mesh = make_mesh(dp=64, tp=8, pp=16)   # Llama-3 405B style

One number to remember: Llama-3 405B trained on 16,000 H100s with TP=8 x PP=16 x CP=2 x DP>=64 — five composed axes to fit one model. The composition is mandatory, not optional.

The rest of this guide is everything that extends or depends on that idea — what each axis costs in memory and bandwidth, how FSDP collapses three of them into one, and how to compose them without leaving the GPUs idle.

---

The fundamental problem

A single H100 has 80 GB of HBM. To train a 70B-parameter model in BF16, you need:

• Weights: 140 GB.
• Gradients: 140 GB (one float per parameter).
• Optimizer state (Adam): 280 GB (two moments per parameter, each in float32 typically — so 8 bytes per parameter).
• Activations: 50–500 GB depending on batch size, sequence length, and whether you're using activation checkpointing.

Total: ~600–1000 GB for one forward+backward pass. None of this fits on one H100.

You have three choices:

1. Reduce memory (smaller model, smaller batch, lower precision, shard state).
2. Use more memory (more GPUs, holding different parts of the work).
3. Trade compute for memory (recompute activations instead of storing them).

Distributed training is the third option turned into engineering: how to spread the work across many GPUs while keeping them busy and synchronized.

---

A short history of distributed training

Distributed training is older than LLMs but the requirements got dramatically harder when language models scaled past a few billion parameters. A timeline:

2012-2014: classical data parallelism. Goyal et al.'s "Accurate, Large Minibatch SGD" (2017) showed how to scale ResNet training to 1024 GPUs with linear-warmup learning rate schedules. The dominant pattern was simple: replicate the model, split the batch, all-reduce gradients. Worked because models fit on a single GPU.

2018-2019: the model-parallel era begins. Megatron-LM (Shoeybi et al., 2019, arXiv:1909.08053) introduced practical tensor parallelism for transformers. Models started exceeding single-GPU memory, requiring the model itself to be split. ZeRO (Rajbhandari et al., 2019, arXiv:1910.02054) introduced sharding optimizer state across data-parallel replicas — the seed of FSDP. Pipeline parallelism arrived in parallel: GPipe (arXiv:1811.06965) and PipeDream (arXiv:1806.03377).

2020: 3D parallelism. Megatron-LM v2 (Narayanan et al., 2021, arXiv:2104.04473) combined DP × TP × PP. NVIDIA's reference architecture for trillion-parameter models. GPT-3 was trained around this era using these techniques.

2021-2022: refinements. ZeRO-3 (full parameter sharding), PyTorch FSDP, GPT-NeoX, and Megatron-DeepSpeed integrations. Training-platform tooling matured. ZeRO-Infinity (arXiv:2104.07857) extended sharding to CPU/NVMe.

2023: long-context training. Llama 2's 4k context required handling activation memory better. Sequence parallelism and selective recomputation (Korthikanti et al., 2022, arXiv:2205.05198) became standard.

2024: 5D parallelism. Llama-3 405B used DP × TP × PP × CP, with FSDP overlay. The combination became normal.

2024-2025: MoE training. DeepSeek-V3 (and others) demonstrated frontier MoE training at scale, requiring careful EP design with all-to-all bisection bandwidth.

2026 (current): 5D parallelism is standard for frontier training. The art is matching parallelism strategy to interconnect topology and model architecture.

The lesson from this timeline: distributed training techniques accumulate, they don't get replaced. Modern frontier training uses every trick from every era simultaneously.

---

Data Parallelism (DP)

The simplest scaling: replicate the model on every GPU. Split the input batch across GPUs. Each GPU does forward+backward on its subset. After backward, all GPUs all-reduce gradients. Then all step the optimizer in lockstep.

GPU 0: model copy A, batch [0:64]
GPU 1: model copy A, batch [64:128]
GPU 2: model copy A, batch [128:192]
...
all_reduce(gradients across all GPUs)
each GPU: optimizer.step()

When DP works

• Model fits on one GPU (after FSDP-style sharding if needed).
• You want more throughput (process more samples per second).
• Communication budget supports the all-reduce.

For a 7B model in BF16 (14 GB weights), a single H100 80GB has plenty of room. Pure DP is the right choice for any setup that fits this profile.

When DP doesn't work

• Model too large to replicate on one GPU. Need TP/PP/FSDP.
• Communication overhead too high (slow interconnect, many GPUs).

DP communication cost

For an N-parameter model, each step's all-reduce moves N × 4 bytes (for float32 grads) per GPU per step. For a 70B model: 280 GB per step.

On NVLink (900 GB/s), the all-reduce on 8 GPUs takes ~30ms. On InfiniBand (200 Gb/s = 25 GB/s), ~10s. The difference is why DP across a node is fine; DP across nodes needs careful overlap with compute or it dominates step time.

Configuring DP

PyTorch DDP:

import torch.distributed as dist
dist.init_process_group("nccl")
model = DDP(model, device_ids=[local_rank])

Multi-node launch:

torchrun --nnodes=4 --nproc_per_node=8 \
  --rdzv_backend=c10d --rdzv_endpoint=$MASTER_HOST \
  train.py

Most modern training stacks (DeepSpeed, FSDP, Megatron-LM) wrap DDP with extra optimizations. Collective performance lives or dies on NCCL — see our NCCL guide and the underlying NVLink and rack-scale topology primer for the hardware context, and AI training networking for the inter-node fabric. Routing tokens across experts adds another wrinkle, covered in mixture-of-experts serving. When you push to FP8/FP4 the rules change again (quantization tradeoffs, NVIDIA datacenter GPUs).

DP and gradient overlap

A subtle but important DDP optimization: overlap the all-reduce with the next layer's backward pass. As gradients become available for layer N, start their all-reduce; while it's flying over the wire, compute backward for layer N-1.

Done well, this hides 50-80% of communication time behind compute. Done badly (e.g., starting all-reduces only after all backward is done), DP communication dominates step time on multi-node setups.

PyTorch DDP's bucket_cap_mb parameter controls how gradients are grouped for all-reduce. Default 25MB. Larger buckets = better bandwidth utilization but worse overlap. Common production value: 50-100MB on fast InfiniBand.

When pure DP fails

The breaking point: model + activations don't fit on one GPU. Once a 7B model + activations + gradients + optimizer state exceeds 80GB, pure DP can't scale further.

The next step is FSDP/ZeRO (shard the redundant state across DP replicas) or TP (split the model across GPUs). Both work; FSDP is simpler, TP is faster at the largest scales.

---

Tensor Parallelism (TP)

Split each weight matrix across GPUs. The classic example: in a transformer's MLP, the up-projection and down-projection matrices are sharded along their inner dimension. Forward computes the first matmul partially on each GPU; an all-reduce combines results before the second matmul.

For attention, the K, Q, V projections are split along the head dimension. Each GPU computes attention for its assigned heads. Output projection requires all-reduce.

When TP works

• Model doesn't fit on one GPU.
• You're within a single node (NVLink-connected).
• Per-layer all-reduce overhead is acceptable (1–10ms per layer on NVLink).

When TP doesn't scale

Across nodes (InfiniBand instead of NVLink), TP all-reduce becomes prohibitive. TP almost never goes past 8 in practice (one node's NVLink fabric).

TP communication cost

Two all-reduces per transformer layer (after attention, after MLP). Each moves batch × seq × hidden × 2 bytes (BF16).

For Llama-3 70B at TP=8 with batch=8, seq=2048, hidden=8192:

• Per all-reduce: 256 MB.
• 80 layers × 2 all-reduces = 160 per forward+backward.
• Total per step: ~40 GB intra-node.

NVLink at 900 GB/s handles this in ~50ms total — small compared to per-step compute time.

Configuring TP

Megatron-LM is the canonical implementation. vLLM, NeMo, DeepSpeed wrap Megatron-style TP. Most users don't write TP code; they configure framework-level flags.

TP requires divisibility

For TP=N to work cleanly:

• num_attention_heads must be divisible by N (e.g., 64 attention heads → TP=8 works, TP=6 doesn't).
• num_kv_heads must be divisible by N (e.g., GQA-8 → TP=8 max).
• hidden_size must be divisible by N for MLP splitting.

Common gotcha: a model with 32 query heads + 4 KV heads (GQA-32) caps TP at 4. Models with num_kv_heads=2 cap at TP=2 — a problem for serving 70B-class models on 8 H100s.

Sequence parallelism (within TP)

A subtle add-on: in Megatron's TP implementation, certain layers (LayerNorm, dropout) are not parallelized but cost activation memory. Sequence parallelism (SP) splits these along the sequence dimension across TP ranks.

Result: ~30% reduction in activation memory at the cost of two extra all-reduces per layer. For long-context training, SP is essential.

TP communication breakdown

Per transformer layer, TP requires:

• Attention output all-reduce: batch × seq × hidden × bytes.
• MLP output all-reduce: same size.

For Llama-3 70B, batch=8, seq=4096, hidden=8192, BF16: each all-reduce is 256 MB.

At 80 layers × 2 all-reduces × 256 MB = 40 GB intra-node per forward pass. NVLink-4 at 900 GB/s handles this in ~50ms. NVLink-5 (B200) at 1.8 TB/s handles it in ~25ms.

Across nodes (IB 400 Gb/s = 50 GB/s): 800ms. Why TP doesn't go cross-node.

---

Pipeline Parallelism (PP)

Split the model by layer across GPUs. Token at position N flows GPU 0 → GPU 1 → ... → GPU N-1 to produce output. Different micro-batches are in flight at different stages simultaneously.

Pipeline bubbles

The naive pipeline has periods where some GPUs idle while waiting for upstream ones. The "bubble" is the idle time.

Two scheduling tricks reduce bubbles:

• GPipe-style: forward all micro-batches, then backward all. Simple but holds activations in memory.
• 1F1B (one-forward-one-backward) (Megatron-LM): interleave forward and backward to overlap. Standard for production training.
• Interleaved 1F1B: split each pipeline stage across multiple non-contiguous layer chunks. Smaller bubbles, more communication.

With 1F1B and 8-stage pipeline, bubble is ~10% of step time. With interleaved 1F1B, ~5%.

When PP works

• Model exceeds what TP+DP can fit.
• You have multi-node infrastructure where TP can't span nodes.
• Communication is point-to-point (slower than all-reduce, doesn't need full bisection bandwidth).

When PP hurts

• Small models — bubbles dominate.
• Very long sequences with PP — activations on each stage become large.
• Inference — bubbles directly increase latency. PP is rarely used for inference.

---

Expert Parallelism (EP)

For Mixture of Experts (MoE) models. Each layer has many "expert" MLPs; each token routes to a small subset (typically top-2 of 8 or top-4 of 64).

EP distributes experts across GPUs. Each token routes via all-to-all communication to its assigned experts, then back.

When EP wins

• MoE models. EP is the natural fit for the architecture.
• Within a node where all-to-all is fast (NVLink).

When EP hurts

• Across nodes — all-to-all has bisection-bandwidth requirements that InfiniBand struggles with at scale.
• Imbalanced routing — if one expert gets routed too much, that GPU bottlenecks.

Modern MoE training

Mixtral, DeepSeek-V3, and similar 2024-2026 MoE models combine EP with TP and PP. DeepSeek-V3 used EP=64 across multiple nodes — possible only because they engineered around all-to-all bottlenecks specifically.

Frontier MoE training uses expert-parallel-aware load balancing — routing decisions consider GPU load, not just token relevance.

EP all-to-all bandwidth math

The bottleneck for EP is the all-to-all collective. For each token, the router decides which experts to send to; tokens are shuffled to their assigned GPUs.

For Mixtral 8×22B with EP=8, batch=8, seq=4096, hidden=8192, top-2 routing:

• Tokens routed: 8 × 4096 = 32,768 per layer.
• Each token routed to 2 experts: 65,536 token-routings per layer.
• Each routing carries hidden_size × bytes: 8192 × 2 = 16 KB.
• Total all-to-all data per layer: 65,536 × 16 KB = 1 GB.

Across 56 layers, ~56 GB of all-to-all traffic per forward pass. On NVLink (900 GB/s): ~60ms total. Acceptable.

Across nodes (IB 400 Gb/s = 50 GB/s): 1.1 seconds. Crippling.

This is why EP almost always stays within a node. Cross-node EP requires either custom hierarchical routing (DeepSeek's approach) or accepting significant slowdown.

Load balancing in MoE training

A subtle problem: routing isn't uniformly distributed across experts. Some experts get more tokens than others. The over-loaded GPU bottlenecks the entire step.

Mitigations:

• Auxiliary loss: penalize unbalanced routing during training. Most MoE training does this.
• Capacity factor: artificially cap how many tokens any expert can see per batch. Excess tokens are dropped (or routed to a fallback expert).
• Token replication: replicate over-utilized experts across multiple GPUs.

DeepSeek-V3's training innovations included sophisticated load balancing that allowed near-uniform routing without the typical auxiliary-loss penalties.

---

Sequence and Context Parallelism (SP/CP)

For training on very long sequences (32k+ tokens), the sequence dimension itself becomes a memory bottleneck. SP and CP split this dimension across GPUs.

Sequence parallelism: split activations along the sequence dimension within a TP group. Used in Megatron-LM. Each TP rank holds 1/N of the sequence's activations during certain layers.

Context parallelism / Ring attention: each GPU holds a contiguous chunk of K and V; ring-style communication passes K/V chunks around so each GPU's local Q can attend to all positions.

When SP/CP matters

• Long-context training (32k+ tokens for billions of tokens).
• Frontier models pretraining at 8k+ context with very large batch.

Combined with TP/PP

Llama-3 405B's training used CP=2 with TP=8 and PP=16 — a 4D parallelism scheme. CP doubled the effective sequence length they could handle without exhausting per-GPU memory.

---

Fully Sharded Data Parallel (FSDP) and ZeRO

FSDP and ZeRO solve the same problem: in pure DP, each GPU stores the full model + gradients + optimizer state. For a 70B model on 64 GPUs that's 64 × 600 GB of redundant state.

The fix: shard model parameters, gradients, and optimizer state across DP ranks.

ZeRO levels (DeepSpeed)

• ZeRO-1: shard optimizer state. Saves the most memory (optimizer state is largest in float32). Lightweight.
• ZeRO-2: shard optimizer state + gradients. More memory savings, slight communication overhead.
• ZeRO-3: shard everything (params, gradients, optimizer). Maximum memory savings, largest communication overhead. Materialize a layer's weights only when needed.

ZeRO-3 is essentially "TP for the entire model with extra steps" but communication is async (overlapped with compute).

FSDP (PyTorch)

PyTorch's native equivalent of ZeRO-3. Each FSDP unit (typically a transformer layer) shards its params across DP ranks. Forward and backward gather params on demand.

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(model, sharding_strategy=ShardingStrategy.FULL_SHARD)

FSDP is simpler than DeepSpeed (more PyTorch-native) but historically had less optimization. By 2026, FSDP-2 has closed most of the gap.

When to use FSDP/ZeRO

• Model too large for pure DP but you don't want TP's complexity.
• Many DP replicas (16+) where the redundancy savings dominate.
• Training that doesn't need TP's per-layer parallelism.

When to combine FSDP with TP

For very large models (100B+), the standard pattern is:

• TP within a node (8-way).
• FSDP across nodes for DP-style scaling with sharded state.

This leverages NVLink for TP's all-reduces and only does sharded-DP-style communication across the slower inter-node links.

Activation memory: the often-forgotten cost

The "memory math for training" formula focuses on weights + gradients + optimizer state + activations. The first three are well-understood. Activations are where most surprise OOMs come from.

Activation memory per layer per token, for a transformer:

• Attention: ~12 × hidden_size × bytes (multiple intermediate tensors saved for backward).
• MLP: ~10 × hidden_size × bytes.

For Llama-3 70B at hidden=8192, BF16, batch=8, seq=4096:

• Per-layer activation: ~12 × 8192 × 2 × 8 × 4096 = 6.4 GB.
• 80 layers: 512 GB total activation memory.

This is larger than the weights. Without activation checkpointing, you can't fit a 70B training run in any reasonable cluster.

Activation checkpointing (selective recomputation)

The fix: don't store all activations. Recompute them during backward pass. Trade compute for memory.

Strategies:

• Full checkpointing: recompute everything. Maximum memory savings, ~30% extra compute.
• Selective checkpointing: only checkpoint expensive-to-store tensors (e.g., post-softmax attention). Standard in Megatron-LM.
• Per-layer checkpointing: checkpoint at layer boundaries. Mid-grain trade-off.

Megatron-LM's selective recomputation (Korthikanti et al., 2022) recomputes the output of softmax-dropout (cheap to recompute, expensive to store). Cuts activation memory by ~75% with only 5% compute overhead.

Activation checkpointing is essential for any training run that doesn't fit in memory. Modern frameworks enable it by default for large models.

Sequence-parallel activation memory

When using sequence parallelism (SP) within a TP group, activations are split along the sequence dimension. This reduces per-GPU activation memory by the SP factor.

For TP=8 with SP enabled, per-GPU activation memory drops to ~1/8 of un-parallelized. Critical for long-context training.

Memory budget worksheet

To plan a training run, work through:

Per-GPU memory budget (e.g., 80 GB on H100):
  Weights (FP8 or BF16) ÷ TP_size: ____
  Gradients (BF16) ÷ TP_size: ____
  Optimizer state (FP32) ÷ DP_size [if FSDP/ZeRO]: ____
  Activations / TP / SP: ____
  CUDA reserved + NCCL buffers: ~5 GB
  Headroom: ~10 GB
  ---
  Total: must fit in 80 GB.

If it doesn't fit: reduce batch size, increase TP/PP, increase activation checkpointing, or upgrade to bigger HBM (H200, B200).

Pipeline parallelism's activation surcharge

PP introduces an extra activation-memory cost: each pipeline stage has to hold activations for all in-flight micro-batches, not just one.

For 1F1B scheduling with N stages and K micro-batches in flight:

• Activation memory per stage: ~K × per-layer-activations × layers_per_stage.

This is why PP doesn't always help — it can multiply activation memory while spreading weights across GPUs. For very long sequences, PP's activation cost can be prohibitive.

Modern PP setups balance this with selective recomputation and careful micro-batch scheduling.

FSDP-2: the rewrite

PyTorch shipped FSDP-2 in late 2024 — a ground-up rewrite that addresses many issues with FSDP-1:

• Better composability with TP (composable.fully_shard).
• Native support for mixed-precision parameters.
• Cleaner gradient and optimizer state handling.
• Easier integration with torch.compile.

For new training runs, FSDP-2 is the right choice. FSDP-1 is still maintained but no longer the default.

FSDP and activation checkpointing

A common combo: FSDP + activation checkpointing. Recompute activations during backward instead of storing them. Trades 30% extra compute for ~50% memory reduction.

from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
    checkpoint_wrapper, CheckpointImpl
)

# Wrap each transformer layer
for layer in model.layers:
    apply_activation_checkpointing(layer)

For frontier-scale training with very long sequences, activation checkpointing is essential. Without it, activations dominate memory at long context.

FSDP eviction patterns

FSDP keeps parameters sharded between forward passes. When forward starts, it all-gathers the parameters for one layer at a time. This creates a sliding window:

Layer 1: gather → forward → free
Layer 2: gather → forward → free
...

The "freed" memory is reused for the next layer. Peak memory is roughly: model size / num_DP_replicas + activations + optimizer state.

If the all-gather can't keep up with forward (slow network), forward stalls waiting for parameters. This is why FSDP across slow networks (no IB, just Ethernet) struggles.

---

3D, 4D, 5D parallelism: combining strategies

Frontier-scale training uses 3+ axes simultaneously. The terminology:

• 3D parallelism: DP × TP × PP. Standard since 2022.
• 4D parallelism: + EP for MoE, or + CP for long-context.
• 5D parallelism: DP × TP × PP × EP × CP. Used in Llama-3 and DeepSeek-V3 training.

Picking the combination

A typical decision tree for 2026 frontier training:

1. Single GPU: pure DP if model fits, FSDP if not.
2. Single node, 8 GPUs, model fits with TP: TP=8.
3. Single node, 8 GPUs, model exceeds: TP=8 + FSDP (sharded).
4. Multi-node, dense model: TP=8 within node + DP across nodes. Add PP if model exceeds TP capacity within a node.
5. Multi-node, MoE model: TP=8 + EP within node + DP across nodes.
6. Long-context training: add CP=2 or CP=4 to whatever else.

Real-world examples

• Llama-3 70B: TP=8 × DP=128 across 16 nodes (1024 H100s).
• Llama-3 405B: TP=8 × CP=2 × PP=16 × DP=64 across many nodes (16,000+ H100s).
• DeepSeek-V3 671B MoE: TP=1 × EP=64 × PP=4 × DP=many across many nodes (2048+ H800s).

The mix isn't arbitrary. It's driven by what fits per-GPU, what scales over the available interconnect, and what the model architecture demands (MoE → EP, long-context → CP).

---

Communication patterns and topology

Different parallelism axes have different communication patterns:

• DP: all-reduce of gradients. Latency-tolerant, bandwidth-heavy.
• TP: all-reduce of activations per layer. Latency-sensitive, very bandwidth-heavy.
• PP: point-to-point send/recv between adjacent stages. Latency-sensitive but lower bandwidth.
• EP: all-to-all per layer. Latency-sensitive, full-bisection bandwidth-heavy.
• CP: ring communication, all-reduce. Sequence-length-dependent.
• FSDP/ZeRO: all-gather of params per layer + reduce-scatter of gradients. Many small collectives.

Topology requirements

Within a node (8 GPUs on NVLink): all collectives are cheap. NVLink-4 = 900 GB/s/GPU. TP, EP, FSDP all fine.

Within a row of nodes (e.g., 32 nodes on a single InfiniBand spine): inter-node bandwidth ~400 Gb/s = 50 GB/s per direction. DP across nodes works with overlap; TP across nodes doesn't.

Across rows (full datacenter): bisection bandwidth limits. Big training jobs are aware of topology and place workers carefully — same-row nodes for DP groups, related ranks on adjacent racks.

This is why you see specifications like "Llama-3 trained on 24,000 H100s in clusters of 1024" — the cluster-of-1024 boundary is the point where TP/EP all-reduce bandwidth scales nicely.

NCCL: the actual primitive layer

All these patterns are implemented via NCCL. See the NCCL guide for tuning.

Misconfigured NCCL is the #1 cause of "training is mysteriously slow." Profile with NCCL_DEBUG=INFO first.

---

Mixed precision training

Training in BF16 (or FP16) instead of FP32 halves memory for weights, gradients, and activations. Doubles tensor-core throughput on Hopper/Ampere GPUs.

The catch: optimizer states usually stay in FP32 (Adam moments need precision for training stability). So you have:

• Weights: BF16
• Gradients: BF16
• Optimizer state (m, v): FP32
• Master copy of weights: FP32 (for optimizer step)

Total per-parameter memory: 2 + 2 + 4 + 8 = 16 bytes. For 70B: 1.1 TB.

FP8 training

Hopper introduced FP8 training. NVIDIA's Transformer Engine library makes this practical. Throughput on H100: ~2× vs BF16.

In 2024-2026 frontier training, FP8 is standard for the MLP layers; attention layers sometimes stay in BF16 due to numerical sensitivity.

See the Mixed Precision Training guide for full details.

---

Gradient accumulation and effective batch size

A 7B-class model trains best at effective batch size 4M tokens. With sequence length 8192, that's 488 sequences per global batch. On 64 GPUs at micro-batch=2, you'd want 488/64/2 = 4 gradient accumulation steps per optimizer step.

Gradient accumulation: forward+backward N micro-batches without stepping the optimizer. Sum gradients across micro-batches. Step once.

for micro_step in range(grad_accum_steps):
    micro_batch = next(dataloader)
    loss = model(micro_batch)
    loss.backward()

optimizer.step()
optimizer.zero_grad()

Effective batch sweet spots

For modern LLM training:

• 7B-class: 4M tokens.
• 70B-class: 8M tokens.
• 400B+: 16M+ tokens.

Smaller batches → more steps to converge, less stable. Larger batches → diminishing returns and increased risk of optimization instabilities.

These come from empirical work (Chinchilla, Llama scaling studies). Treat 4–16M as a starting range.

Hyperparameter scaling laws

A key practical question: when you scale up a training run, how do you adjust learning rate, batch size, weight decay, etc.?

Learning rate: scales roughly inversely with sqrt(batch_size). Doubling batch size requires multiplying LR by ~1.4×. The "Chinchilla optimal LR" papers cover the math.

Warmup steps: typically 0.5-1% of total training steps. Longer warmup helps stability for very large batches.

Weight decay: stays roughly constant (0.1 typical) across model sizes. Some recent work suggests slightly higher for larger models.

Gradient clipping: typically 1.0. Helps prevent gradient explosions, especially in mixed-precision training.

Adam betas: 0.9 / 0.95 (β1 / β2) for most LLM training. Lower β2 makes the second-moment estimate more reactive — sometimes helps stability for very large models.

Scheduling: cosine learning rate decay is the modern default. Linear warmup → cosine decay to ~10% of peak LR.

For pre-training a new model, start with reference numbers from a similar-scale published model (Llama, Mistral, DeepSeek). Adjust based on early-step loss behavior.

Loss curve health monitoring

The loss curve tells you the most about training health. Things to watch:

Smooth decline: healthy training has a smooth, monotonically decreasing loss with mild noise. The shape is logarithmic (drops fast, then plateau-y).

Loss spikes: sudden upticks. Usually transient (one bad batch). If they happen >once per 1000 steps, investigate.

Divergence: loss starts rising, doesn't recover. Causes: gradient explosion (check grad norms), bad data (a corrupted batch), hyperparameter mismatch.

Plateaus: loss flattens earlier than expected. May indicate: data quality issue, learning rate too low, model capacity bottleneck.

Train/val divergence: train loss decreasing but validation loss flat or rising. Overfitting. Reduce model size, add regularization, increase data.

Track these continuously during training. Most setups plot loss live on Weights & Biases or similar.

Gradient norms

Track L2 norm of gradients per layer. Healthy training: gradient norms are O(1), relatively stable across steps.

Pathologies:

• Gradient norms growing unbounded → gradient explosion. Lower learning rate, increase grad clipping.
• Gradient norms collapsing to zero → vanishing gradients. Check for dead neurons, learning rate too low.
• Layer-specific spikes → that layer is unstable. Lower learning rate for it, or initialize differently.

---

Checkpointing and resharding

A 70B-parameter model checkpoint is ~140 GB (BF16) + ~280 GB (optimizer state in FP32). For a model trained with 5D parallelism, the checkpoint is split across many shards.

Sharded checkpoints

Each GPU rank writes its own shard. Loading requires the same parallelism layout, or a resharding step.

Modern frameworks (PyTorch's distributed.checkpoint, NeMo, DeepSpeed) save in formats that are resharded-on-load — you can take a checkpoint trained with TP=8 × PP=4 × DP=64 and resume with TP=4 × PP=8 × DP=128.

Frequency

Frontier training writes checkpoints every 1000–5000 steps. That's typically every 1–4 hours of wallclock. The cost: 5–15 minutes of training time per checkpoint.

Some setups use async checkpointing — copy state to host memory in a background thread while training continues. Reduces stall to seconds.

Storage

A 70B checkpoint × N versions can quickly become petabytes. Modern training stacks tier checkpoints to object storage (S3, GCS) with local NVMe cache for the most recent.

---

Fault tolerance at scale

A 10,000-GPU training run will lose a GPU every ~hour. At frontier scale, hardware failures are not edge cases.

Mitigations:

Checkpointing: standard. The basic backstop.

Replicated distributed state: ZeRO-style sharding implicitly replicates state across DP ranks. A failed GPU's shard can be recovered from peers within the DP group.

Hot spares: standby GPUs that take over when a peer fails, requiring only a state transfer rather than a full restart.

Asynchronous training: in some setups, workers can continue with stale gradients while a failed worker is replaced.

The frontier labs (Meta, Google, Anthropic, OpenAI) have invested heavily in fault tolerance. Llama-3's training had a "mean time to recovery" of minutes, not hours.

---

The major frameworks

Megatron-LM (NVIDIA)

The reference for TP and PP. Most production frontier training uses Megatron or a fork. Strengths: most optimized TP/PP/CP, NVIDIA's first-party support. Weaknesses: complex API, NVIDIA-specific.

DeepSpeed (Microsoft)

ZeRO and pipeline parallelism. Strengths: comprehensive memory optimizations, broad model support. Weaknesses: complex configuration.

FSDP (PyTorch)

Native PyTorch alternative to ZeRO. Strengths: simple API. Weaknesses: TP support requires extra layers.

NeMo (NVIDIA)

NVIDIA's higher-level framework. Wraps Megatron with sane defaults. Strengths: easier to start. Weaknesses: less flexible for unusual setups.

Lightning (Lightning AI)

Pythonic distributed training. Strengths: clean API, good for research. Weaknesses: not the fastest at frontier scale.

Pick by use case

• Frontier-scale training, NVIDIA-only: Megatron-LM.
• Research / experiments: PyTorch FSDP or Lightning.
• Production training pipelines: NeMo or DeepSpeed.
• MoE-specific: DeepSpeed or Megatron-LM with EP support.

Megatron vs DeepSpeed in detail

The two most-used frameworks for frontier-scale training. They have different design philosophies.

Megatron-LM (NVIDIA):

• Bottom-up. Implements TP, PP, CP from first principles in CUDA.
• Tightly coupled to PyTorch but with custom kernels.
• Optimized specifically for transformer models.
• Best raw performance on dense models.
• Steeper learning curve; more configuration knobs.

# Megatron-LM config (simplified)
megatron_args = {
    "tensor_model_parallel_size": 8,
    "pipeline_model_parallel_size": 4,
    "context_parallel_size": 2,
    "sequence_parallel": True,
    "use_distributed_optimizer": True,  # Megatron's ZeRO-1 equivalent
}

DeepSpeed (Microsoft):

• Top-down. ZeRO is the centerpiece; pipeline and tensor parallelism layered on.
• More accessible API.
• Better for ZeRO-3 / FSDP-style sharded training.
• MoE support.
• ZeRO-Infinity for offloading params/grads to CPU/NVMe (rare but useful).

{
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {"device": "cpu"}
  },
  "tensor_parallel": {"tp_size": 8},
  "pipeline_parallel": {"pp_size": 4}
}

When to pick each:

• Megatron-LM: dense transformer training at frontier scale. Best peak performance.
• DeepSpeed: when ZeRO-style sharding is the dominant strategy. Easier to start.
• Megatron-DeepSpeed (combined): many production setups use both. Megatron for TP/PP/CP, DeepSpeed for ZeRO. NVIDIA NeMo wraps both.

NeMo as a higher-level abstraction

NVIDIA's NeMo framework wraps Megatron with more sane defaults and pre-built recipes. For most teams starting in 2026, NeMo is the easier on-ramp:

from nemo.collections.llm import api as llm
recipe = llm.gpt(model_size="70B")
recipe.trainer.devices = 64
recipe.trainer.tensor_model_parallel_size = 8
recipe.trainer.pipeline_model_parallel_size = 4
trainer = nl.Trainer(**recipe.trainer.dict())

NeMo handles checkpointing, recovery, telemetry, and most operational concerns. Trade off some flexibility for production-readiness.

Lightning Fabric

PyTorch Lightning's distributed training abstraction. Less feature-rich than Megatron/DeepSpeed but cleaner API:

from lightning.fabric import Fabric
fabric = Fabric(accelerator="cuda", devices=64, strategy="fsdp")
fabric.launch()
model, optimizer = fabric.setup(model, optimizer)

Good for research experiments. Not the right choice for frontier training.

---

Capacity planning a training run

How many GPUs do you need? The math.

Inputs

• Model size (parameters).
• Target compute (tokens × params).
• Time budget (days/weeks).
• Hardware (H100, H200, B200).

Procedure

1. Compute total training FLOPs: Chinchilla-optimal is ~20 tokens per parameter, ~6× FLOPs per token-parameter pair. For 70B model × 1.4T tokens: 5.9×10²² FLOPs.

2. Compute peak FLOPs/sec available: H100 = 1979 TFLOPs/sec FP16. Realistic sustained: 40-50% = ~700 TFLOPs/sec.

3. Compute total GPU-hours: 5.9×10²² / (700×10¹² × 3600) = 23,000 GPU-hours.

4. Pick wallclock time: 30 days = 720 hours. So 32 GPUs minimum.

5. Pick parallelism: 70B with TP=8 in BF16 fits on one node. So 4 nodes of 8 H100s = 32 GPUs, DP=4 × TP=8.

6. Add overhead for fault tolerance: ~10% extra capacity.

Worked example: pretraining a 7B from scratch

Target: 7B params, 1.5T tokens, 14 days wallclock.

• Total FLOPs: 7B × 1.5T × 6 = 6.3×10²².
• GPU-hours at H100 FP8: 11,700.
• Wallclock 14 days = 336 hours. So 35 GPUs minimum.
• Round to 32 H100s (4 nodes × 8 GPUs). Pure FSDP, batch size 4M tokens.
• Add 4 spare GPUs for fault tolerance: 36 H100s for 14 days.

Worked example: 405B frontier-scale training

Target: 405B params, 15T tokens, 90 days.

• Total FLOPs: 405B × 15T × 6 = 3.6×10²⁵.
• GPU-hours at H100 FP8 (50% utilization): 6.7M.
• Wallclock 90 days = 2160 hours. So 3100 H100s.
• Add 30% for fault tolerance + checkpointing overhead: ~4000 H100s.

This roughly matches the 16,000 H100 setup Meta used for Llama-3 405B. Their setup achieved higher utilization than 50%, so they reached the same training in fewer absolute GPU-hours but with much faster wallclock (54 days).

---

Compiler-level optimizations

Beyond the parallelism strategy, training throughput depends on how efficiently the model code maps to GPU instructions.

torch.compile

PyTorch 2.0+ ships torch.compile, which JIT-compiles model code into optimized CUDA kernels. For training, the win is typically 10-30% throughput.

model = torch.compile(model, mode="max-autotune")

Modes:

• default: balanced compile time and runtime gains.
• reduce-overhead: minimal compile time, modest gains.
• max-autotune: longest compile time, maximum gains.

Caveats:

• First step is much slower (compilation).
• Some PyTorch idioms break compilation (Python conditionals, custom autograd).
• Debugging is harder — errors point to compiled code, not original Python.

For frontier training, torch.compile is usually worth it. Validate on a small run first.

CUDA Graphs

Capture a sequence of CUDA operations once and replay them. Eliminates Python and CUDA dispatch overhead.

Used heavily in Megatron-LM for the inner training loop. Most production training uses CUDA Graphs implicitly via the framework.

Custom kernels

For specific operations (attention, RMSNorm, MoE routing), custom CUDA kernels can outperform PyTorch's general-purpose ops by 20-50%.

The major implementations:

• FlashAttention (Tri Dao): the canonical attention kernel.
• xformers (Meta): collection of memory-efficient ops.
• Triton (OpenAI): Python-like DSL for writing GPU kernels.
• Liger Kernel (LinkedIn, 2024): fused kernels for common LLM ops.

Modern training stacks integrate these by default. You rarely write custom kernels yourself.

Fused operations

Many separate ops can be fused into one kernel: residual + add + layer norm + linear all in one CUDA invocation. Reduces memory bandwidth (intermediate tensors stay in registers).

Megatron-LM's apex.optimizers.FusedAdam is a common example — fuses all of Adam's operations into one launch.

Selective vs full activation checkpointing trade-off

no checkpointing:    100% memory, 100% compute
full checkpointing:  ~30% memory, ~130% compute
selective:           ~50% memory, ~105% compute

The selective version is usually right. Full checkpointing's compute overhead can offset its memory savings if you're already compute-bound.

---

Gradient compression for slow networks

For training across geographically-separated clusters or over slow inter-node links, gradient compression can reduce communication cost.

Techniques

Top-k: only send the K largest gradients per layer per step. Recover others later. ~1000× compression possible; quality cost is small if K is tuned right.

Random-k: similar but random subset. Simpler but slightly lower quality.

1-bit Adam: quantize Adam's update to 1 bit per element. ~32× compression on the optimizer step's communication.

Gradient sparsification with momentum compensation: explicitly track and compensate for skipped gradients in the next step.

These are mostly used in academic settings or for federated learning. Frontier training within a single datacenter has fast enough networks that compression isn't needed.

For the rare case of cross-datacenter training (DiLoCo, federated): essential.

---

Curriculum learning and data scheduling

Modern frontier training uses sophisticated data scheduling:

Sequence-length curriculum

Start training with short sequences (1k-2k); progressively increase to target context length (8k-32k+) over the course of training.

Why: shorter sequences are faster per step. Early-stage learning happens on shorter context; later refinement uses long context.

Llama-3 used this pattern. Llama-3.1 extended to 128k context via length-curriculum continued training.

Difficulty curriculum

Train on "easier" data first (simpler text), then progressively harder (technical, code, reasoning). Some evidence this helps convergence; not universal.

Domain weighting

Different proportions of different data sources (web vs code vs books) over training. Some setups dynamically adjust based on which sources are most informative at each stage.

DeepSeek's papers describe their data weighting strategies in detail.

Tokens per epoch

Frontier training in 2026: 1-2 trillion tokens, single pass (one epoch). Smaller training runs may use multiple epochs (2-4 typical).

Going past ~4 epochs typically hurts more than helps — the model memorizes data instead of generalizing.

---

Post-training: SFT, DPO, RLHF, RLVR

Pre-training produces a base model. Post-training turns it into something useful. The major stages:

Supervised Fine-Tuning (SFT)

Train on (prompt, ideal response) pairs. Standard task-specific loss (next-token prediction).

Distributed setup: typically full DP, since SFT models fit comfortably on one node. For 70B SFT: 8-16 GPUs.

Compute: ~1-5% of pre-training compute. Cheap by comparison.

DPO and variants (Direct Preference Optimization)

Train on (prompt, preferred response, rejected response) triples. Optimize the model to prefer the preferred response without explicit reward modeling.

DPO is much cheaper than RLHF (no separate reward model, no rollouts). It's become the standard "post-training" technique for many open-weight models.

Distributed setup: same as SFT. The loss is just a different formula on the same data structure.

RLHF (Reinforcement Learning from Human Feedback)

The classic post-training technique. Three stages:

1. SFT on demonstrations.
2. Train a reward model on human preferences.
3. RL fine-tune the model against the reward model.

Distributed setup: significantly more complex. Need distributed rollouts (model generates trajectories), centralized reward computation, distributed policy updates.

Frameworks: TRL (Hugging Face), TRL-X, RLlib variants. NeMo-RL is NVIDIA's frontier-scale offering.

RLVR (Reinforcement Learning with Verifiable Rewards)

Used for o1/R1-style reasoning models. Rewards come from verifiable signals (correct math answer, working code, passing test cases) rather than human preferences.

Distributed setup: similar to RLHF but with code-execution or proof-checking infrastructure. The reward computation can be expensive (running test cases takes wall-clock time).

OpenAI's o1 and DeepSeek's R1 are products of RLVR-style training.

How post-training fits in the pipeline

A modern frontier model goes through:

1. Pre-training (months, $50-200M).
2. SFT (days, ~$100k).
3. DPO/RLHF/RLVR (weeks, ~$1-5M).
4. Evaluation, iterative refinement.
5. Release.

Steps 2-4 take 1-5% of pre-training compute but are critical for quality. Frontier labs invest heavily in post-training infrastructure.

---

Emerging training techniques

The field is moving. Watch:

MoE training improvements

DeepSeek-V3's load-balancing innovations cut MoE training overhead significantly. Expect more progress here through 2026-2027.

FP4 training

Native FP4 training on Blackwell. Quality data is preliminary but promising. By 2027, expect FP4 to be standard for forward-pass MLPs.

Architecture-specific optimizers

Adam was designed for general optimization. New optimizers (Lion, Sophia, Shampoo) are tuned for transformer training. Some show 10-20% improvement.

Distillation pipelines

Frontier models distill capabilities into smaller models. Standard for 7B/13B-class production models. Llama-3.2 1B/3B were distilled from Llama-3 70B.

Efficient context-extension

Llama-3.1 went from 8k to 128k context via continued training. Techniques (NTK-aware scaling, YaRN, Long-RoPE) make this much cheaper than training from scratch at long context.

Federated frontier training

Training across multiple datacenters or organizations. DiLoCo and follow-ups make this technically possible. Still slower than centralized; mainly relevant for compliance scenarios.

Sparse attention training

Native Sparse Attention (NSA) and similar enable longer-context training without quadratic compute. Becoming standard for 1M+ context targets.

Custom silicon

Google's TPUs, Cerebras CS-3, SambaNova RDU — alternatives to GPUs. Training on these requires different distributed-training strategies. Performance is competitive for specific workloads.

---

Cost economics

Frontier training costs 8 figures and up.

Indicative numbers (mid-2026)

• H100 lease: ~$2-4/hour on cloud, $1-2/hour on dedicated reserved.
• H200: ~$3-5/hour.
• B200: ~$5-8/hour (early 2026 pricing).

For 11,700 GPU-hours (small frontier-scale 7B): $25,000-50,000. For 6.7M GPU-hours (Llama-3 405B equivalent): $15M-25M.

Hidden costs

• Networking: InfiniBand fabric for 1000+ GPU clusters costs $1000+ per port. A 16,000-GPU cluster has tens of millions in networking alone.
• Storage: petabytes of training data + checkpoints.
• Idle time during failures: ~5% of training time is "stalled or restarting."
• Engineering: a frontier training team is 20-50 people. $5M+/year salaries.

The total cost of a Llama-3 405B-scale training run, end-to-end, is in the $50M-$200M range. Foundation labs treat this as capex.

---

Failure modes

NCCL hangs

A single failed GPU or network blip causes a collective to hang. All workers freeze.

Fix: NCCL_TIMEOUT, watchdogs, automatic restart policies. Modern training stacks have all of this.

Loss spikes

Loss suddenly increases, sometimes diverges. Causes: gradient explosions, bad data, hyperparameter mismatches, mixed precision overflow.

Fix: gradient clipping, loss scaling, checkpoint and roll back if it doesn't recover.

Checkpoint corruption

A failed write produces a corrupt checkpoint. Resuming from it crashes mysteriously.

Fix: write-then-rename atomic semantics, multiple checkpoint copies, validation on save.

Stragglers

One node is 10% slower than others. The whole job runs at the slowest node's pace.

Fix: detect stragglers via per-rank timing, replace slow nodes proactively. Some setups use "elastic" training that drops slow nodes mid-run.

Tokenizer mismatch on resume

You changed the tokenizer between training runs. Resumed checkpoint sees different token IDs.

Fix: pin tokenizer version. If you change it, retrain from scratch.

Data ordering bugs

Data loader produces different batches across runs (non-deterministic shuffling, race conditions). Reproducibility breaks.

Fix: deterministic data loading, fixed random seeds for shuffle.

Real failure case studies

Case 1: Llama-3 70B run that diverged at step 60,000

Symptom: loss had been declining smoothly for 60,000 steps, then sudden spike followed by NaN.

Diagnosis: a single corrupted training file in the data mix. The mix included a CommonCrawl shard that was duplicated billions of times in a single file due to a preprocessing bug. When the data loader hit it, the model saw the same text repeatedly within a batch, which caused gradient explosion.

Fix: rolled back to step 55,000 checkpoint; deduplicated the data; resumed.

Lesson: validate training data quality. Spot-check batch contents periodically.

Case 2: 405B training that took 1.5× expected time

Symptom: training was hitting expected throughput in synthetic benchmarks but real training was slow.

Diagnosis: the training data loader was IO-bound. Reading from network storage at 200 MB/s vs the 2 GB/s the GPUs could consume.

Fix: pre-staged data to local NVMe; data loader read locally. Throughput jumped to expected rate.

Lesson: benchmark the data path, not just the compute path.

Case 3: gradient explosion from a single bad layer

Symptom: gradient norms growing monotonically over 5000 steps. Training continued but quality plateaued.

Diagnosis: a specific transformer layer had unusual initialization that caused activations to grow over training.

Fix: re-initialized that layer with smaller variance; resumed from earlier checkpoint.

Lesson: track gradient norms per layer, not just globally.

Case 4: tokenizer mismatch on resume

Symptom: training resumed cleanly but loss was 5× higher than where it left off.

Diagnosis: an environment update changed the tokenizer version. Previously-tokenized batches had different IDs.

Fix: reverted tokenizer; pinned version explicitly.

Lesson: pin every dependency that affects data semantics.

Common anti-patterns

Mistakes I've seen teams make repeatedly:

Over-engineering parallelism: trying to use TP=8 × PP=8 for a 7B model. Pure DP would be fine. Parallelism strategies have overhead; only use what's needed.

Under-using hardware: running TP=2 when TP=4 would fit the model with more headroom. Wastes per-GPU memory and limits batch size.

Not gradient-clipping: training without gradient clipping. Single bad batches can destabilize training.

Using fp16 instead of bf16: FP16 needs loss scaling and is more brittle. Use BF16 unless your hardware doesn't support it.

Skipping warmup: launching training without LR warmup. Causes early-step instability.

Insufficient checkpointing: checkpointing every 10,000 steps. A single failure can cost hours. Modern recommendation: 1000-2000 steps.

Ignoring the data loader: assuming the GPU is the bottleneck. Often the data loader is. Profile both.

Not validating reproducibility: never confirming that a re-run produces the same loss curve. Bugs accumulate silently.

Running in float32: training in pure FP32. 4× slower than necessary on Hopper+.

Single point of failure for checkpoints: storing checkpoints on one server with no replication. One disk failure = lost training run.

Distributed training cheat sheet

A quick reference for picking parallelism:

Model fits on 1 GPU?
├── Yes → Pure DP. Use DDP or FSDP if you have many replicas.
└── No → Need model parallelism.
    ├── Fits with TP=8 in one node? → TP=8.
    │   └── Multi-node? Add DP across nodes.
    │       └── State too large for replicated DP? → FSDP-2.
    ├── Doesn't fit with TP=8? → Add PP.
    │   └── Across multiple nodes.
    └── MoE model? → Add EP across experts.

Long context (>32k)?
└── Add CP or activation checkpointing.

Going for max throughput on NVIDIA?
└── Megatron-LM with custom kernels.

Need flexibility?
└── DeepSpeed with ZeRO-3 or PyTorch FSDP-2.

This is the decision tree most production teams follow.

---

Megatron-LM TP, SP, and CP in detail

Megatron-LM is the canonical reference implementation of tensor parallelism and pipeline parallelism. Its sequence parallelism and context parallelism extensions are the standard for long-context training. Understanding what Megatron actually does is the foundation for understanding most frontier training stacks.

Tensor parallelism in Megatron

Megatron's TP partitions weight matrices across GPUs along specific dimensions. For an attention layer with QKV projection, the projection matrix is split along the output dimension (column-parallel); for the output projection, split along the input dimension (row-parallel). The combination produces an all-reduce at the end of attention rather than at every step.

For MLP layers, the first matmul (the up-projection) is column-parallel; the second matmul (the down-projection) is row-parallel. Same all-reduce pattern at the end.

The structural insight: by alternating column-parallel and row-parallel matmuls, Megatron minimizes communication to one all-reduce per layer rather than at every matmul.

Sequence parallelism

Megatron's sequence parallelism is a refinement of TP that addresses memory overhead in layer norms and dropouts. In standard TP, every rank holds the full activation tensor for layer norms (because layer norm operates across the hidden dimension, which is partitioned). With SP, activations are partitioned along the sequence dimension during layer norm, then all-gathered for the matmul, then re-partitioned. The result: dramatically lower peak activation memory, at the cost of an additional all-gather and reduce-scatter per layer.

SP is essentially free in compute (the all-gather and reduce-scatter overlap with adjacent matmuls) but saves substantial memory. Most production Megatron deployments use SP by default with TP.

Context parallelism

For very long contexts (32K+, in some setups 1M+), even SP's per-layer memory becomes a bottleneck because attention is O(sequence^2) in compute and memory. Context parallelism partitions the sequence dimension across GPUs and computes attention with explicit ring-style communication.

The Megatron CP implementation uses a ring-attention-style pattern: each rank holds 1/N of the sequence, exchanges KV blocks with neighboring ranks, and accumulates partial attention outputs. The communication pattern overlaps with computation; for sufficiently long sequences the throughput hit is modest (10-20%).

DeepSpeed-Ulysses

A different approach to long-context attention is DeepSpeed-Ulysses, which partitions across the head dimension rather than the sequence dimension. The trade-off: Ulysses scales with the number of attention heads (fixed by the model), CP scales with sequence length (variable). Ulysses works well for short-to-medium context with many heads; CP works well for very long contexts.

Most production frontier training stacks support both and choose based on the workload. For background on the attention mechanics behind these methods, see long-context attention.

---

DeepSpeed ZeRO stages 1, 2, 3, Infinity

DeepSpeed's ZeRO is the other major lineage of distributed training, alongside Megatron. The stages refer to what's partitioned across data-parallel ranks.

ZeRO-1: optimizer state partitioning

Each rank holds the full model weights and gradients but only 1/N of the optimizer state. Memory savings: 4x for a model trained with Adam (the optimizer state is the largest single piece of training memory). Communication overhead: minor (the optimizer step is parallelized).

ZeRO-1 is essentially "DDP with optimizer state shared." Cheap memory win.

ZeRO-2: gradient partitioning

Builds on ZeRO-1 by also partitioning gradients. Each rank holds 1/N of the gradients. The gradient all-reduce becomes a reduce-scatter (each rank's portion of the gradient is collected on its rank). Memory savings: ~8x over DDP for Adam. Communication: same as ZeRO-1.

ZeRO-2 is the right default for moderate-scale training. Most teams running 7-70B fine-tuning on a single node use ZeRO-2.

ZeRO-3: parameter partitioning

Builds on ZeRO-2 by also partitioning the model parameters themselves. Each rank holds 1/N of the parameters. During forward pass, parameters are all-gathered just before they're needed and discarded after. During backward, the gather-discard cycle repeats.

Memory savings: full linear scaling with the number of ranks. A 70B model in BF16 with FP32 master and Adam state is around 1.4TB; with ZeRO-3 across 16 ranks, each rank holds about 90GB. Communication: substantially higher than ZeRO-2 (parameters are gathered twice per step instead of once).

ZeRO-3 is the basis for PyTorch FSDP.

ZeRO-Infinity: NVMe offload

Extends ZeRO-3 by offloading parameters, gradients, and optimizer state to NVMe SSD when not actively in use. The CPU and GPU memory hold only the slice currently in compute; the rest sits on disk.

The tradeoff: throughput drops significantly (3-5x slower than ZeRO-3) because NVMe is much slower than HBM. The use case is training models too large to fit in any reasonable cluster's aggregate GPU memory. For most teams, ZeRO-Infinity is a "I cannot afford a bigger cluster" workaround, not a default choice.

Per-stage memory math

For a 70B model in BF16 weights + BF16 gradients + FP32 master + FP32 Adam (m and v):

Stage	Per-rank memory (16 ranks)	Per-rank memory (256 ranks)
DDP	1.4TB	1.4TB
ZeRO-1	950GB	750GB
ZeRO-2	600GB	350GB
ZeRO-3	90GB	5.5GB
ZeRO-Infinity	30GB (rest on NVMe)	4GB

ZeRO-3 (and equivalently FSDP) is what makes frontier-scale training memory-feasible. Without it, a 70B model would not fit on a 256-GPU H100 cluster.

---

FSDP1 vs FSDP2: the PyTorch 2.6 rewrite

PyTorch FSDP has gone through a major architectural revision. FSDP1 (the original implementation) and FSDP2 (the PyTorch 2.4+ rewrite, stable as of PyTorch 2.6) differ in important ways.

FSDP1: the flat-parameter model

FSDP1 organized parameters into "FlatParameter" groups — each module's parameters were concatenated into a single contiguous buffer, sharded across ranks, and unsharded on demand. The flat-parameter design enabled efficient all-gather and reduce-scatter but had limitations: it couldn't handle modules with mixed precision cleanly, it had difficulty with selective freezing, and the abstraction leaked through to user code in confusing ways.

FSDP2: per-parameter sharding via DTensor

FSDP2 uses PyTorch's DTensor (distributed tensor) abstraction. Each parameter is a DTensor sharded according to a ParallelMesh specification. The sharding is per-parameter rather than per-module, which fixes the limitations of FSDP1:

• Mixed-precision parameters: each parameter can have its own dtype.
• Selective freezing: freeze any parameter without disrupting the sharding.
• Composability with TP: a parameter can be sharded along the FSDP dimension and the TP dimension simultaneously.

The downside: FSDP2 is somewhat slower than FSDP1 for simple workloads because the DTensor abstraction adds per-parameter overhead. For most modern workloads the trade is worth it.

The ParallelMesh API

FSDP2 introduces ParallelMesh as the abstraction for declaring multi-dimensional parallelism. A user defines a mesh (e.g., 2D mesh with 4 TP ranks and 8 DP ranks) and applies sharding plans to parameters along each mesh dimension. The mesh API generalizes to higher-dimensional parallelism cleanly.

The mental model: ParallelMesh is the canonical way to express "this parameter is sharded along these dimensions of the parallelism plan." Everything else (FSDP, TP, expert parallelism in MoE) becomes a specific sharding plan applied to parameters via the mesh.

Migration from FSDP1 to FSDP2

For typical training scripts, the migration is straightforward — replace FSDP1 wrappers with FSDP2 calls and the rest of the code unchanged. For custom training loops with non-trivial sharding logic, the migration requires more careful work because FSDP2's per-parameter model is fundamentally different.

The 2026 status: FSDP2 is the recommended default in PyTorch 2.6+. FSDP1 is maintained for backward compatibility but receives no new features.

FSDP2 + TP composition

The killer feature of FSDP2 is its composition with tensor parallelism. A 70B model can be sharded along both the FSDP (data-parallel) dimension and the TP (tensor-parallel) dimension simultaneously, using a 2D mesh. The combination produces a clean, frameworkable expression of 3D parallelism that was awkward in FSDP1.

For more depth on 3D parallelism patterns, see the existing section on combining DP, TP, and PP earlier in this guide.

---

Pipeline schedules: GPipe, 1F1B, interleaved 1F1B, Zero Bubble

Pipeline parallelism partitions layers across GPUs. The pipeline schedule (the order in which forward and backward passes execute across stages) determines throughput and bubble overhead.

GPipe

The original pipeline schedule. All forward passes complete before backward begins. Simple, but produces a large "bubble" of idle time at the start (warm-up) and end (cool-down) of each minibatch. Bubble overhead is roughly (P - 1) / (P + M) where P is the number of pipeline stages and M is the number of micro-batches.

1F1B (one-forward-one-backward)

After the warm-up phase, the schedule alternates between one forward pass and one backward pass per stage. Reduces the bubble compared to GPipe while keeping the schedule simple. Standard in Megatron-LM and DeepSpeed.

Interleaved 1F1B

Divides each stage's layers into multiple chunks; each rank handles multiple chunks. The pipeline becomes deeper (more stages logically) which reduces the bubble overhead proportionally to the interleaving factor. The cost is more pipeline communication.

Megatron-LM's interleaved 1F1B with interleaving factor 4 typically reduces the bubble by 3-4x relative to standard 1F1B. The communication overhead is a 10-20% increase.

Zero Bubble (ZB-H1, ZB-H2)

A 2024 series of schedules that further reduce the pipeline bubble by overlapping forward and backward computation more aggressively. ZB-H1 reduces the bubble to near-zero by splitting the backward into "backward through inputs" and "backward through weights" with careful scheduling. ZB-H2 extends the optimization with additional memory cost.

The schedules are implemented in some production stacks (Megatron-LM, some Tülu pipelines) but not yet universal. The throughput gain is meaningful (5-15% over interleaved 1F1B for typical workloads) but the implementation complexity is real.

Schedule comparison

Schedule	Bubble overhead	Implementation complexity	Memory cost	When to use
GPipe	~30-50% (typical)	Low	Low	Educational, never production
1F1B	~15-25%	Medium	Medium	Standard default
Interleaved 1F1B	~5-10%	High	Medium	Most production frontier training
Zero Bubble	~0-3%	Very high	High	Cutting-edge, throughput-critical

The pattern: more sophisticated schedules trade implementation complexity for less bubble overhead. The choice depends on the team's engineering depth and the value of the throughput recovery.

---

Memory math worked example: Llama 70B

A concrete walkthrough of memory accounting for a 70B model trained with BF16 weights, FP32 master, Adam optimizer, and FSDP2.

Per-rank memory components

For a 70B model on a 64-GPU cluster with FSDP2 sharding (64-way data parallelism):

• Model weights (BF16): 70B params * 2 bytes = 140GB total, 2.2GB per rank.
• Master weights (FP32): 70B * 4 bytes = 280GB total, 4.4GB per rank.
• Gradients (BF16): 70B * 2 bytes = 140GB total, 2.2GB per rank.
• Adam first moment (FP32): 280GB total, 4.4GB per rank.
• Adam second moment (FP32): 280GB total, 4.4GB per rank.

Sharded total: 17.6GB per rank for the model state.

Activation memory

Activation memory depends on batch size and sequence length. For batch size 1024 globally (16 per rank), sequence length 8192, 80 transformer layers:

• Per-layer activation: roughly 4 * batch * seq * hidden * 2 bytes = 4 * 16 * 8192 * 8192 * 2 = 8.6GB.
• With activation checkpointing (recompute on backward, store at boundaries): roughly 1GB per checkpoint boundary.
• With selective activation checkpointing (store some activations, recompute others): tunable trade-off.

For full activation checkpointing: 80 layers * 1GB = 80GB. Without checkpointing: 80 * 8.6GB = 688GB (infeasible).

Total per-rank memory: 17.6GB (model state) + 80GB (activation checkpointed) = 97.6GB. Fits on H200 141GB; tight on H100 80GB.

What changes with FP8

FP8 weights cut the weight bytes in half: 70B * 1 byte = 70GB total, 1.1GB per rank. Master weights stay FP32 (4.4GB). Adam state stays FP32 (8.8GB). Per-rank model state: 14.3GB instead of 17.6GB. Modest savings.

The bigger FP8 win is throughput (1.8-2.0x), not memory.

What changes with gradient accumulation

Gradient accumulation runs multiple micro-batches and accumulates gradients before the optimizer step. Memory cost: the accumulation buffer (FP32 typically). For 16-way accumulation and 70B model: 280GB total accumulation buffer, 4.4GB per rank.

The pattern: gradient accumulation trades extra memory for the ability to use larger effective batch sizes. For most production training the trade is worthwhile.

What changes with longer context

At 32K sequence length instead of 8K, activations grow by 4x. With activation checkpointing, the per-checkpoint boundary memory grows by 4x. Total activation memory becomes 320GB at full checkpointing — too much for H100 80GB.

The fix is context parallelism (partition the sequence across additional ranks) or DeepSpeed-Ulysses. Both effectively reduce per-rank activation memory at the cost of inter-rank communication.

---

DiLoCo for cross-DC training

Most of this guide assumes a single cluster connected by InfiniBand or NVLink. For training across multiple datacenters or geographically distributed providers, the bandwidth assumptions fail and different methods are needed.

Why cross-DC training is hard

A cluster has ~400-3200 GB/s of cross-node bandwidth (InfiniBand HDR/NDR). A cross-DC link might have 10-100 GB/s of bandwidth. The all-reduce volume that's negligible on a single cluster becomes the dominant cost across DCs.

For a 70B model, the gradient all-reduce per step is around 140GB in BF16. On a single cluster, that completes in ~0.5 seconds. Across a 10 GB/s WAN link, it takes 14 seconds — likely longer than the actual compute step.

The DiLoCo solution

DiLoCo (DeepMind, 2023) addresses this by running many local SGD steps without inter-DC communication, then synchronizing via a slow outer-loop averaging. Inter-DC bandwidth drops by approximately 500x.

The math: each DC runs 500 inner SGD steps locally (no inter-DC comm). After 500 steps, DCs all-reduce the "outer gradient" (difference between current weights and starting weights). The inter-DC all-reduce happens once per 500 steps instead of once per step.

What DiLoCo costs

Convergence is slower than synchronous training in wall-clock for the same FLOPs. Typical penalty: 1.1-1.5x more steps to reach the same final loss. This is much less than the bandwidth savings, so the overall wall-clock can still be better than synchronous training across the bandwidth-limited link.

OpenDiLoCo and Prime Intellect

Prime Intellect's open-source implementation (OpenDiLoCo, 2024) and their actual cross-continent training runs (Paris-SF training of 1B and 10B models in 2024-2025) are the most credible production demonstrations. The 10B model trained on commodity internet links reaches loss curves competitive with single-cluster training.

For frontier-scale (70B+) training, DiLoCo is not yet competitive with centralized clusters. The hope is that the next generation of methods (DisTrO and successors) close that gap.

For deeper coverage of the decentralized context, see decentralized GPU compute.

---

Reference designs at 100, 1000, 10000 GPU scale

Putting it all together: what an actual production training stack looks like at three different scales.

100-GPU design (single rack)

Typical setup: 8 nodes of 8x H100 SXM each, NVLink within node, InfiniBand HDR (200 Gbps) between nodes, single InfiniBand switch.

Training stack:

• Model size: 7B-30B dense, 30B-50B with aggressive sharding.
• Parallelism: FSDP2 across all 64 ranks; no pipeline or tensor parallelism.
• Precision: BF16 with optional FP8 for the matmul path.
• Communication: NCCL with default tuning; the cluster is small enough that defaults work.
• Framework: PyTorch FSDP2 + Transformer Engine, or DeepSpeed ZeRO-3.

Wall clock for a 7B pretraining run on 1T tokens: roughly 30-50 days. Total cost: $150K-$300K.

1000-GPU design (multi-rack)

Typical setup: 125 nodes of 8x H100/H200 SXM, fat-tree InfiniBand topology with 800 Gbps per node, rail-optimized routing.

Training stack:

• Model size: 70B dense, 200B MoE.
• Parallelism: 2D parallelism — FSDP2 within rails, TP across rails (Megatron-style).
• Precision: FP8 with Transformer Engine HYBRID recipe.
• Communication: Tuned NCCL (NCCL_IB_QPS_PER_CONNECTION, topology-aware ring), see NCCL guide.
• Framework: Megatron-LM + Transformer Engine, or NeMo, or a heavily customized PyTorch stack.

Wall clock for a 70B pretraining run on 2T tokens: roughly 30-50 days. Total cost: $1.5M-$3M.

10000-GPU design (frontier cluster)

Typical setup: 1250 nodes of 8x H100/H200/B200, multi-rail InfiniBand topology with 3200 Gbps aggregate per node, hierarchical switch design, dedicated checkpoint storage with multi-TB/s aggregate bandwidth.

Training stack:

• Model size: 400B+ dense, 1T+ MoE.
• Parallelism: 3D or 4D parallelism — DP + TP + PP, possibly + EP for MoE.
• Precision: FP8 with block-wise scaling (DeepSeek-style).
• Communication: Heavily tuned NCCL plus custom communication patterns for specific layers.
• Framework: Custom in-house or heavily-customized Megatron + Transformer Engine.
• Storage: Distributed file system with checkpoint sharding; see checkpoint storage and recovery.
• Fault tolerance: Automatic restart from latest checkpoint on any node failure; node-failure rate at this scale is 1-3 per day.

Wall clock for a 405B pretraining run on 15T tokens: roughly 60-90 days. Total cost: $30M-$80M.

What scales linearly and what doesn't

Throughput scales sub-linearly with GPU count due to communication overhead. The MFU (model FLOPs utilization) drop from 100 to 10000 GPUs is typically 20-40% (e.g., 50% MFU at 100 GPUs, 30-40% MFU at 10000 GPUs). The drop is dominated by inter-rack communication latency and the increasing complexity of the parallelism plan.

Frontier labs spend substantial engineering effort to maintain MFU above 40% at 10K+ scale. The discipline pays back in months of wall-clock saved.

For the underlying GPU and networking specs, see NVIDIA datacenter GPUs and AI training networking.

---

The bottom line

The named problem is the memory wall: a single GPU cannot hold a frontier model, and the only way out is to split state across many GPUs without leaving them communication-stalled. The solution is composed parallelism — DP, TP, PP, EP, SP, and FSDP each slice a different dimension of training state, and frontier runs stack 3-5 of them on a multi-axis device mesh. The single biggest lever is picking which axis takes which physical interconnect — TP belongs on NVLink, DP on InfiniBand, PP across racks, EP wherever all-to-all is cheapest.

What to do if you take only this away:

• If your model fits on one GPU: plain DDP or FSDP with ZeRO-2 is enough. Don't reach for tensor parallelism.
• If it doesn't fit but fits on one node: TP up to the NVLink boundary, then DP outward.
• If it doesn't fit on one node: add PP across nodes (one stage per node), keep TP intra-node, DP across replicas.
• For MoE: add EP equal to the number of routed experts per token; watch all-to-all latency.
• For >32k context: add SP/CP; ring attention is the standard primitive.
• Always profile communication overlap before adding another axis — an idle GPU is more expensive than an extra all-reduce.

Next, read collective communication for AI training for the NCCL behaviour each axis depends on, and NVIDIA datacenter GPUs for which SKU's memory and NVLink topology decides your TP bound.

---

FAQ

Q: When should I use FSDP vs Megatron?

FSDP is simpler, PyTorch-native. Megatron is faster at the largest scales. For 7B-class training: FSDP is fine. For 70B+: Megatron wins. For 400B+: definitely Megatron.

Q: What's the difference between DDP and FSDP?

DDP replicates the model on every GPU; FSDP shards it. DDP is fine for models that fit on one GPU; FSDP is for models that don't.

Q: How big a batch size should I use?

For modern LLM training, 4M tokens for 7B-class, 8M for 70B-class, 16M+ for 400B+.

Q: Should I train in BF16 or FP8?

BF16 is the safe default. FP8 doubles throughput but is less forgiving — needs careful loss scaling and may fail on numerically sensitive layers.

Q: How do I debug a slow training run?

Profile first. NVIDIA Nsight Systems shows per-GPU and per-collective timing. Common bottlenecks: NCCL misconfig, slow data loader, stragglers, OOM thrashing.

Q: How do I handle GPU failures during training?

Modern frameworks have built-in resume from checkpoint. Frontier setups also have fault-tolerant variants that don't even fully restart.

Q: Should I use spot instances for training?

For research/experimentation, yes — checkpointing handles preemption. For frontier training with strict deadlines, dedicated hardware is more predictable.

Q: When does context parallelism become necessary?

For sequence lengths beyond what your TP+PP can fit. Typically 32k+ tokens with ~70B+ models. Below that, just use TP+PP.

Q: What's the minimum cluster size to train a 70B model?

Roughly 16 H100s minimum (TP=8 × DP=2). 32 H100s gives reasonable wall-clock time for fine-tuning. Pre-training a 70B from scratch needs 256+ GPUs to finish in a reasonable time.

Q: How do I budget time for a training run?

Empirical formula: tokens × 6× FLOPs per token-parameter-pair, divided by sustained throughput. Sustained throughput is ~40-50% of peak FLOPs. For a 70B model on 64 H100s training 1.4T tokens: ~2 weeks.

Q: What's the difference between continuous batching and gradient accumulation?

They're orthogonal concepts. Gradient accumulation is a training technique (sum gradients over N micro-batches before stepping). Continuous batching is a serving technique (admit new requests into the in-flight batch).

Q: Why does my training run plateau early?

Common causes: data quality (your dataset has fundamental issues), model capacity (the model is too small for the task), learning rate schedule (decayed too aggressively), data exhausted (you're seeing each example >1 epoch).

Q: How do I know if my parallelism strategy is right?

Profile a training step. If communication time is < 30% of step time, you're well-tuned. If > 50%, your parallelism strategy is too aggressive — reduce TP or PP.

Q: What's the right train/eval split for LLM pretraining?

99.9% / 0.1% is standard. Eval set should be in-distribution but never seen during training. Sample of 1000-10000 examples is enough for tracking loss curves.

Q: How do I migrate from FSDP-1 to FSDP-2?

Mostly drop-in. Update the import path, change the wrapping API. Some configurations differ; check the migration guide. FSDP-2 has cleaner state dict handling, so checkpoint resharding is easier.

Q: Should I use Megatron-DeepSpeed or pure Megatron?

Megatron-DeepSpeed is a NVIDIA + Microsoft collaboration that combines Megatron's TP/PP with DeepSpeed's ZeRO. Use it if you need both. Pure Megatron is simpler if you don't need ZeRO-3.

Q: How does training cost scale with model size?

Roughly quadratically in compute (parameters × tokens × 6). Tokens scale roughly linearly with parameters (Chinchilla-optimal). So total cost scales as parameters² approximately. Training a 700B model is ~100× the cost of a 70B model.

Q: What about training on Apple Silicon or AMD?

Apple Silicon: not viable for serious training. Single-node, no NVLink-equivalent for multi-GPU.

AMD: increasingly viable. PyTorch + ROCm + DeepSpeed work on MI300/MI350. Performance is competitive with H100 for many workloads. Ecosystem maturity (Megatron, NeMo) lags NVIDIA.

Q: How do I pre-train a custom model architecture?

Start with an existing architecture (Llama, Mistral) and modify. Reuse the data pipeline, optimizer, and most of the training infrastructure. The model code is the smallest part of the work.

Q: What's the standard data mix for LLM pretraining in 2026?

Web crawl (CommonCrawl, FineWeb, RefinedWeb) is the bulk. Plus code (GitHub, StackOverflow), books, academic papers (PubMed, arXiv), Wikipedia, and curated specialty datasets. Specific mixes vary; FineWeb-Edu is a strong baseline for high-quality data.

Q: Should I use synthetic data?

Increasingly common. Llama-3.1's training included substantial synthetic data. Generation cost is low; quality control matters. Don't replace organic data; supplement it.

Q: How do I evaluate a partially-trained model?

Standard benchmarks: MMLU, GSM8K, HumanEval, HellaSwag, ARC. Run periodically (every 1000-5000 steps). Watch for: loss correlation with downstream metrics (sometimes diverges), benchmark contamination (leakage from training data).

Q: What about distributed RL for post-training?

Rapidly evolving area. Common pattern: distributed rollouts (many GPUs generate trajectories) + centralized policy update. Frameworks like NeMo-RL and TRL handle the orchestration.

Q: How do I handle very long training runs (months)?

Frequent checkpointing (every 1000 steps), distributed checkpointing (each rank writes its shard), automated restart on hardware failures, monitoring infrastructure. Frontier labs invest heavily here.

Q: What's the role of model parallelism in fine-tuning?

For fine-tuning, you usually need much less parallelism than pre-training. A 70B model can be fine-tuned on 8 H100s with TP=8 + LoRA. Full fine-tuning needs more memory (16-32 GPUs typical).

Q: How do I evaluate distributed training infrastructure?

Step 1: nccl-tests on the cluster. Verify achieved bandwidth matches expected.

Step 2: end-to-end training run on a small model. Measure tokens/sec, GPU utilization.

Step 3: longer run (1000+ steps). Watch for memory leaks, NCCL hangs, throughput drift.

Step 4: failure injection — kill a worker mid-training. Verify automatic recovery.

Step 5: compare to published numbers (Llama paper, NVIDIA reference). If you're 2× slower, something's off.

Q: What's the difference between fault tolerance and elasticity?

Fault tolerance: training survives one or more failures, recovers transparently.

Elasticity: training can dynamically add or remove workers without restart.

Modern frontier training has fault tolerance. Elasticity is rarer (most jobs have fixed worker count).

Q: How do I budget engineering time for a training infrastructure?

Rough estimate: 1 senior engineer-year per 1000 GPUs of cluster scale.

Smaller clusters (16-100 GPUs): 1-2 part-time engineers can manage.

Frontier clusters (10k+ GPUs): dedicated team of 5-20 engineers.

Costs are ops, debugging, optimization, framework upkeep. Doesn't include the training research itself.

Q: What about training stability over very long runs?

For 90-day training runs, infrastructure stability is critical:

• Fault tolerance must work reliably.
• Monitoring must catch issues early.
• Checkpoint integrity must be guaranteed.
• Recovery procedures must be tested.

Frontier labs invest heavily here. A failed training run costs $50M+.

Q: How do I migrate from PyTorch DDP to FSDP?

Mostly drop-in for the model:

# Before: DDP
model = DDP(model)

# After: FSDP
model = FSDP(model, sharding_strategy=ShardingStrategy.FULL_SHARD)

Caveats: optimizer needs awareness of sharded params; some custom layers need FSDP wrapping. Plan for a few days of integration.

Q: What's the relationship between distributed training and data engineering?

Tight. Data must be available faster than the GPUs can consume it.

For 1024 H100s training a 70B model: ~32 GB/s of input data. Need data loaders that can sustain this.

Data engineering is often the unsung hero. Pipelines, distributed shuffling, prefetching all matter.

Q: How do I handle very small training runs (single GPU)?

Don't bother with distributed training tooling. Just single-GPU training with PyTorch's default.

If memory is tight, use FSDP-style sharding within a single device (gradient checkpointing).

Q: What's the typical training:eval split for LLMs?

Most pretraining: 99.9% train, 0.1% eval. Eval set is small — just for tracking loss curves.

For domain-specific fine-tuning: 95% train, 5% eval. Larger eval set helps detect overfitting.

Q: Will compilers eventually replace explicit parallelism?

Active research. PyTorch's torch.distributed, Triton, JAX's pjit all aim to abstract parallelism.

In 2026, explicit parallelism is still standard. Auto-parallelism is improving but not yet at the level where it matches hand-tuned configs.

By 2028-2029: maybe.

Q: What's the training time for various model sizes (rough)?

On 64 H100s, FP8, Chinchilla-optimal data:

• 1B params: 2-3 days.
• 7B params: 1-2 weeks.
• 70B params: 6-12 weeks.
• 400B params: 6-12 months on much bigger clusters.

These are starting points; actual time depends on cluster details.

Q: How important is initialization for distributed training?

Important but solved. Modern initialization schemes (Xavier, He) work well at any scale.

For very deep models, careful initialization (e.g., scaled by depth) matters. Standard implementations handle this.

Q: What tools do I need for distributed training profiling?

• NVIDIA Nsight Systems: per-GPU timeline.
• PyTorch Profiler: per-operation breakdown.
• TensorBoard / Weights & Biases: training metrics.
• Prometheus + Grafana: cluster-level metrics.
• Custom logging: NCCL_DEBUG, framework-specific.

For frontier-scale: invest in profiling infrastructure. Catches issues early.

Q: How do I scale the data pipeline?

Distributed data loaders. Sharded datasets. Prefetching ahead of compute.

Common patterns:

• Webdataset (sharded tar files).
• Mosaic Streaming (efficient streaming).
• Custom pipelines built on PyTorch DataLoader.

For frontier: each GPU has dedicated data shards, loaded in parallel. Streaming from object storage with local caching.

Q: Should I use spot instances for training?

For research/experimentation: yes. Save 50-70% on cost.

For production training with strict deadlines: no. Spot preemptions disrupt the run.

Tools like SkyPilot help orchestrate spot+on-demand mixes for cost-tolerant training.

Q: What's the future of distributed training in 2027+?

Likely directions:

• Auto-parallelism (compiler picks strategy).
• More aggressive use of sparse architectures (MoE, NSA).
• Federated training across organizations.
• Specialized silicon (more TPUs, more custom chips).
• FP4 standard for forward pass.

The fundamentals (DP/TP/PP/EP) will persist. Implementation details will evolve.

---

Q: When should I use HuggingFace Accelerate?

Accelerate is the right answer when you want a unified abstraction over DDP, FSDP, DeepSpeed, and Megatron without committing to one. It's particularly strong for SFT and DPO workflows where you're iterating on configurations. For frontier-scale training the framework's overhead and limitations become bottlenecks; teams typically migrate to Megatron-LM or a custom stack.

Q: When should I use Lightning Fabric?

PyTorch Lightning's Fabric is a lower-level entry into the Lightning ecosystem. Useful when you want Lightning's logging and checkpointing primitives but more control over the training loop. Comparable to Accelerate in scope; the right choice depends on which ecosystem your team already uses.

Q: Is Colossal-AI worth using?

Colossal-AI is a research-tier framework with strong implementations of several parallelism techniques (heterogeneous training, gradient compression, custom communication patterns). Used by smaller research teams that want capability beyond Accelerate but don't need Megatron-scale. In 2026 it's a niche choice; most teams default to PyTorch FSDP2 or Megatron.

Q: What does MosaicML Composer add over native PyTorch?

Composer (now part of Databricks) wraps the training loop with a callback-driven API and includes implementations of many efficiency tricks (selective activation checkpointing, ALiBi-style attention, etc.). The value is integration: trying a new technique is a flag rather than a rewrite. The trade-off is the abstraction overhead and Databricks-specific quirks.

Q: Is Ray Train a viable training framework?

Ray Train is a higher-level orchestration on top of PyTorch DDP, FSDP, or DeepSpeed. The Ray value is cluster orchestration and fault tolerance, not the training algorithm itself. Teams that use Ray for the rest of their ML platform often use Ray Train; teams that don't have Ray usually don't need to add it just for training.

Q: What's the difference between ZeRO-3 and FSDP?

Conceptually nothing — they implement the same algorithm (partition parameters, gradients, and optimizer state across data-parallel ranks). FSDP is PyTorch-native; ZeRO-3 is DeepSpeed-native. Implementation details differ (memory layout, communication scheduling, integration with other parallelism). For new projects, FSDP2 is the recommended default in 2026.

Q: How does FSDP2 compose with TP?

Via PyTorch's ParallelMesh API. Define a 2D mesh (e.g., 8 TP ranks x 16 FSDP ranks for a 128-GPU cluster), apply TP sharding along one dimension and FSDP sharding along the other. The composition is cleaner in FSDP2 than in FSDP1 because the per-parameter DTensor model handles the 2D sharding natively.

Q: When does pipeline parallelism become worth the engineering complexity?

When the model doesn't fit even with FSDP3 + TP, or when the TP communication is consuming too much wall clock. For 70B dense models on H100, FSDP2 + TP is usually sufficient; PP becomes necessary at 200B+ dense or at very large MoE configurations.

Q: What's the right gradient accumulation factor?

Set it so the effective batch size (per-step batch size * gradient accumulation factor * world size) matches the learning rate schedule's target. For typical transformer training, the target effective batch size is 1-4M tokens per step. Gradient accumulation is the knob to hit that target when your per-step batch is too small.

Q: How do I detect a straggler node?

Track per-rank step time; a node consistently slower than its peers is a straggler. The cause is usually hardware (failing fan, throttling GPU, network port at half-speed). Most production stacks have automatic straggler detection and node rotation. Without it, run a per-rank timing dump every N steps and alert on outliers.

Q: Can I mix BF16 and FP8 in the same training run?

Yes, and this is the standard pattern. Most production frontier training keeps numerically-sensitive layers (LayerNorm, softmax, sometimes the LM head) in BF16 while running MLP and attention matmuls in FP8. Transformer Engine and other FP8 implementations expose per-layer precision control.

Q: What's a reasonable target for MFU at my cluster scale?

For 100-GPU clusters: 50-55% MFU is achievable. For 1000-GPU clusters: 40-45%. For 10K+ clusters: 35-40%. These targets assume well-tuned communication and a reasonable parallelism plan; significantly lower numbers suggest tuning work is possible.

Q: How long does a typical 70B pretraining run take in 2026?

On a tuned 1024-GPU H100 cluster with FP8: roughly 30-45 days for 2T tokens of training. On a BF16-only stack: roughly 50-70 days. On older A100 hardware: 90-120 days. The H100-to-H200 transition adds maybe 20% throughput; H100-to-B200 adds 80-100%. Hardware generation matters enormously.

Q: What's the cheapest way to train a 7B model in 2026?

Pretrain from a strong base (Llama, Qwen, Mistral) rather than from scratch — the foundation model already has 1-2T tokens of pretraining you don't need to redo. SFT a strong base for $5K-$30K on a decentralized network; full pretraining from scratch on a curated dataset would cost $200K-$1M for comparable quality. The economics strongly favor adaptive use of existing base models.

Q: How do I budget for an MoE training run?

MoE training is more memory-intensive per active parameter but cheaper per total parameter. A 200B-active MoE costs roughly the same to train as a 70B dense model for the same training token budget. The cost discipline that makes MoE attractive is the inference cost — see mixture-of-experts serving — not the training cost.

---

Distributed training tooling comparison

A side-by-side of the major frameworks for production use.

Megatron-LM

Dimension	Megatron-LM
TP support	Best in class
PP support	Best in class
FSDP/ZeRO equivalent	Distributed Optimizer
MoE support	Yes (full EP)
FP8 support	Best in class
Documentation	Comprehensive but dense
Community	NVIDIA-led, growing
Best for	Frontier-scale dense training

DeepSpeed

Dimension	DeepSpeed
TP support	Full
PP support	Full
ZeRO	Best in class
MoE support	Yes
FP8 support	Yes (via TE)
Documentation	Good
Community	Microsoft-led, large
Best for	ZeRO-style training, MoE

PyTorch FSDP

Dimension	FSDP-2
TP support	Improving (FSDP-2)
PP support	Limited
Sharding	Best for native PyTorch
MoE support	Limited
FP8 support	Via torchao
Documentation	Excellent
Community	PyTorch-native
Best for	Research, ZeRO-style training

NeMo

Dimension	NeMo
TP/PP/CP	Wraps Megatron
ZeRO	Wraps DeepSpeed
MoE	Yes
FP8	Yes
Documentation	Recipe-driven
Best for	Production training pipelines

Lightning

Dimension	Lightning
API ergonomics	Best
Performance at scale	Behind
Best for	Research, experimentation

For most teams: NeMo for production, Megatron for max performance, FSDP for research, DeepSpeed for ZeRO-specific needs.

---

A worked example: setting up a Llama-3 70B fine-tune

End-to-end recipe for fine-tuning Llama-3 70B on 32 H100s.

Hardware and environment

• 4 nodes × 8 H100 SXM each.
• NVLink within node, NDR InfiniBand between nodes.
• Ubuntu 22.04, CUDA 12.4, NCCL 2.21+.
• PyTorch 2.4+, Megatron-LM or NeMo.

Configuration

# 32 GPUs total
# TP=8 within node, DP=4 across nodes
trainer = NeMoTrainer(
    devices=8,
    num_nodes=4,
    strategy=NeMoMegatronStrategy(
        tensor_model_parallel_size=8,
        pipeline_model_parallel_size=1,
        sequence_parallel=True,
    ),
    precision="bf16-mixed",
    accelerator="gpu",
)

# Data: instruction-following dataset
data_module = NeMoDataModule(
    train_data="path/to/train.jsonl",
    val_data="path/to/val.jsonl",
    micro_batch_size=2,
    global_batch_size=128,
    seq_length=4096,
)

Memory math

• Llama-3 70B BF16 weights: 140 GB / TP=8 = 17.5 GB per GPU.
• Adam state FP32 (2× weights size): 35 GB per GPU.
• Activations (with selective recomputation): ~10 GB per GPU.
• Total: ~62 GB per GPU. Fits in 80 GB H100 with headroom.

Training command

torchrun --nnodes=4 --nproc_per_node=8 \
    --rdzv_backend=c10d --rdzv_endpoint=$MASTER:29500 \
    train.py \
        --model-config llama-3-70b \
        --data-config instruction-tuning \
        --trainer-config 32gpu-bf16

Expected throughput

For a 70B fine-tune on 32 H100s:

• Tokens/sec/GPU: ~5,000.
• Aggregate: ~160,000 tok/sec.
• Time per epoch: ~1.5 hours for 1B-token dataset.
• Total fine-tune: 3-7 days for typical setups.

Validation

Periodic eval every 500 steps:

• MMLU subset (~500 questions).
• Custom benchmarks for the fine-tune target.

Watch for: loss decreasing smoothly, eval accuracy increasing or stable, no gradient norm explosions.

Failure handling

• Auto-resume on NCCL hang (timeout 30 min).
• Auto-checkpoint every 500 steps.
• Auto-retry single-replica failures.

Cost estimate

32 H100s × 5 days × $4/hr = $15,360. Reasonable for a meaningful fine-tune.

This recipe is the template most production fine-tunes follow.

Q: What's a good train/eval split for LLM pretraining?

Most frontier setups use 99.9% train / 0.1% eval. The eval set is small enough not to hurt training but big enough to give meaningful loss tracking.

Q: How do I add a new dataset to my training mix?

Carefully. Data mixing is highly empirical. Modern training uses dynamic data weighting (some datasets seen more than others) tuned via small ablations.

Q: Can I train on AMD MI300/MI350?

Possible, increasingly mature. PyTorch + ROCm + DeepSpeed can do it. Performance is competitive with H100 in 2026 for many workloads. NVIDIA's ecosystem maturity (Megatron, NeMo) doesn't fully exist on AMD yet.

---

Glossary

• DP: Data Parallelism. Replicate model, split batch.
• TP: Tensor Parallelism. Split each weight matrix.
• PP: Pipeline Parallelism. Split model by layer.
• EP: Expert Parallelism. Distribute MoE experts.
• CP / SP: Context / Sequence Parallelism.
• FSDP: Fully Sharded Data Parallel. PyTorch's ZeRO equivalent.
• ZeRO: DeepSpeed's memory-sharding scheme.
• All-reduce: collective that sums and broadcasts.
• All-to-all: collective where every rank sends data to every other rank.
• Pipeline bubble: idle time in pipeline parallelism while waiting for upstream.
• Micro-batch: small batch processed in one forward+backward.
• Gradient accumulation: summing gradients across micro-batches.
• NCCL: NVIDIA's collective communication library.
• Megatron-LM: NVIDIA's reference TP/PP implementation.
• DeepSpeed: Microsoft's training framework with ZeRO.
• Mixed precision: training with some operations in lower precision.

---

References

Foundational research

• Megatron-LM (TP) — Shoeybi et al., 2019. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." arXiv:1909.08053. The canonical tensor-parallel transformer implementation.
• ZeRO — Rajbhandari et al., 2019. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." arXiv:1910.02054. Sharded optimizer / gradient / parameter state — the seed of FSDP.
• 3D parallelism on GPU clusters — Narayanan et al., 2021. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." arXiv:2104.04473. The DP × TP × PP recipe used for GPT-3-scale runs.
• GPipe — Huang et al., 2018. "GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism." arXiv:1811.06965. Synchronous pipeline parallelism with micro-batching.
• PipeDream — Narayanan et al., 2018. "PipeDream: Fast and Efficient Pipeline Parallel DNN Training." arXiv:1806.03377. Asynchronous 1F1B scheduling — the basis of modern interleaved pipelines.
• Reducing activation recomputation — Korthikanti et al., 2022. "Reducing Activation Recomputation in Large Transformer Models." arXiv:2205.05198. Selective recompute + sequence parallelism — now standard in Megatron.
• ZeRO-Infinity — Rajbhandari et al., 2021. "ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning." arXiv:2104.07857. Offloads optimizer state to CPU/NVMe.
• PyTorch FSDP — Zhao et al., 2023. "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel." arXiv:2304.11277. The PyTorch-native ZeRO-3 used by most 2024+ training stacks.
• Mixed-precision training — Micikevicius et al., 2017. arXiv:1710.03740. FP16 + dynamic loss scaling.
• FP8 formats — Micikevicius et al., 2022. arXiv:2209.05433. E4M3/E5M2 — the numerics used by Transformer Engine.
• Ring attention — Liu et al., 2023. "Ring Attention with Blockwise Transformers for Near-Infinite Context." arXiv:2310.01889. Context parallelism for million-token training.

Production systems

• Llama 3 technical report — Meta, 2024. arXiv:2407.21783. 405B run details: 5D parallelism, networking, checkpointing.
• DeepSeek-V3 technical report — DeepSeek-AI, 2024. arXiv:2412.19437. MoE training with FP8, DualPipe, and aggressive overlap.
• Chinchilla — Hoffmann et al., 2022. "Training Compute-Optimal Large Language Models." arXiv:2203.15556. Sets the compute/tokens ratio that drives modern training budgets.

Background reading

• NCCL — NVIDIA Collective Communications Library. Documentation.
• PyTorch distributed overview — pytorch.org docs.
• DeepSpeed — Microsoft Research. deepspeed.ai.

---

Memory math worked end-to-end

The single highest-value exercise for any distributed-training engineer is computing per-GPU memory before launching a job. Skipping this is the #1 cause of OOM-after-hour-of-training pain. Here is the full formula.

The per-GPU memory budget

For a transformer with P parameters, sequence length S, batch size B, hidden size H, layers L, on an 80 GB H100, using BF16 weights/gradients + FP32 optimizer states + selective activation recomputation:

weights_per_gpu       = (P × 2) / TP / FSDP_shard_factor
gradients_per_gpu     = (P × 2) / TP / FSDP_shard_factor
optimizer_per_gpu     = (P × 12) / TP / FSDP_shard_factor   # m + v + master copy
activations_per_gpu   = ~(B × S × H × layers_per_stage × 12) / TP / SP
cuda_overhead         = 3 GB
nccl_buffers          = 1 GB per channel × ~8 channels = 8 GB
total                 = sum above

For Llama 70B (P=70B, H=8192, L=80) at B=2, S=4096, with TP=8, PP=2, FSDP across 8 DP replicas:

• Weights: 70B × 2 / 8 / 8 = 2.2 GB.
• Gradients: 2.2 GB.
• Optimizer: 70B × 12 / 8 / 8 = 13.1 GB.
• Activations (40 layers per stage, selective recompute): 2 × 4096 × 8192 × 40 × 12 / 8 = ~4 GB.
• CUDA + NCCL overhead: ~11 GB.
• Total: ~32.5 GB per GPU. Comfortable on 80 GB.

Without FSDP sharding the optimizer, it would be 70B × 12 / 8 = 105 GB. Doesn't fit. This is why FSDP/ZeRO-1 is universal for 70B+ training.

Why activations are the sneaky bit

The above formula uses selective recomputation from Korthikanti et al. — only the cheap-to-recompute, expensive-to-store activations are checkpointed. Without selective recompute, activations balloon by ~4×. Without any activation checkpointing, they balloon by another 4×.

For an 8k-context, 64-batch training run, full activations of Llama 70B can exceed 1 TB per replica. Activation engineering is at least as important as weight engineering.

When OOM still happens

The formula above is a lower bound. Real OOMs happen because of: (1) PyTorch allocator fragmentation — set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True; (2) loss-scaling buffers in FP8; (3) collective scratch space that's larger than the buffer above; (4) gradient unscaling FP32 master copies you didn't account for. Always reserve 10-15 GB of headroom.

---

Communication-computation overlap deep dive

A 50% performance swing hides behind whether your collectives overlap with your compute. This is what separates a 35% MFU run from a 55% MFU run.

What overlap means concretely

When the GPU is computing layer N's matmuls on its compute SMs, the NVLink and IB transports can be moving layer N-1's gradients (DDP) or layer N+1's parameters (FSDP). Both happen on different hardware engines — they only conflict on HBM bandwidth.

How to verify overlap in practice

Profile a training step with NVIDIA Nsight Systems. In the timeline, look for: NCCL kernels on one stream, compute kernels (cuBLAS, FlashAttention) on another stream, both active simultaneously. If they're serialized — one waits for the other — overlap is broken.

The most common cause of broken overlap is calling .item() or print() on a tensor mid-step, which forces a host-device sync that flushes all streams.

DDP gradient overlap

DDP buckets gradients (bucket_cap_mb=25 by default) and starts AllReduce when a bucket is full. The compute-collective overlap window is the time between when the bucket fills and when the next bucket needs it. Smaller buckets → more overlap opportunities but more collective overhead. Larger buckets → less overlap but more efficient per-collective.

Sweet spot empirical: 25-50 MB on intra-rack IB; 100 MB on cross-rack or slow networks where collective overhead dominates.

FSDP parameter prefetch

FSDP-2's fully_shard with MixedPrecisionPolicy and explicit prefetch can overlap layer N+1's AllGather with layer N's compute. Without prefetch, AllGather happens just-in-time and stalls compute.

Enable: fsdp_wrap_policy = ModuleWrapPolicy({TransformerBlock}) and compile=True so the compiler reorders the prefetch.

Pipeline parallel overlap (1F1B)

In 1F1B scheduling, stage N's forward of micro-batch K runs in parallel with stage N+1's forward of micro-batch K-1. The bubble shrinks to (stages - 1) / micro_batches. For 16 stages and 64 micro-batches, bubble is ~23% — bad. For 16 stages and 256 micro-batches, ~6% — acceptable.

Interleaved 1F1B (Megatron) splits each stage into chunks (e.g., 4 chunks of 5 layers instead of 1 chunk of 20 layers), cutting the bubble further at the cost of more inter-stage communication.

Async optimizer step

For ZeRO-1 / FSDP, the optimizer step can overlap with the next forward pass. PyTorch 2.2+'s optimizer_in_backward does this — gradient computation triggers optimizer step for that parameter immediately, freeing the gradient buffer. Saves ~10% wall-clock for parameter-heavy models.

---

Picking parallelism for your model size

A decision matrix for the most common configurations.

Model	GPUs	Recommended config	Why
7B fine-tune	8× H100	Pure DP + LoRA, or FSDP-2 full	Fits per-GPU, no model parallelism needed
7B pre-train	32× H100	FSDP-2 + selective recompute	DP scales linearly with simple comm
13B fine-tune	16× H100	FSDP-2 + LoRA	26 GB BF16 needs sharding
70B fine-tune	32× H100	TP=8 + DP=4, FSDP-1 on DP	Fits with TP=8 single-node
70B pre-train	256-1024× H100	TP=8 × PP=2 × DP=many + FSDP-1	Wall-clock scales DP
70B w/ 32K context	64× H100	TP=8 × CP=2 × DP=4	CP for activation memory
405B fine-tune	64× H100	TP=8 × PP=4 × DP=2	Weights need PP
405B pre-train	4096-16000× H100	TP=8 × PP=16 × CP=2 × DP=many	Llama-3 recipe
Mixtral 8×22B	32× H100	TP=4 × EP=8 + DP	MoE needs EP
DeepSeek 671B	1024+× H100	EP=64 × TP=1 × PP=4 × DP	EP scales experts

The pattern: dense models add TP first, PP second, FSDP for memory. MoE models add EP. Long context adds CP.

---

Cross-DC and federated training

Training across datacenters is moving from research curiosity to production reality in 2026.

Why teams attempt it

(1) Single-DC power constraints — even 100 MW facilities can't host frontier training in one building. (2) Data sovereignty — regulations require training data stays in a region. (3) Cost optimization — buy spot capacity across multiple providers. (4) Resilience — DC outages no longer halt training.

The challenge

Inter-DC latency is 5-50 ms; bandwidth is 100 Gbps-10 Tbps. Both are 10-100× worse than intra-DC IB. Standard synchronous DP doesn't tolerate this — gradient AllReduce latency multiplies into step time.

DiLoCo and async DP

DiLoCo (DeepMind, 2023) trains local copies at each DC, synchronizes weights every 500-1000 steps via slow but rare global AllReduce. Effective bandwidth requirement drops by ~500×. Quality cost: 1-3% loss penalty depending on tuning. Used in production by Prime Intellect, Nous Research, and others in 2025-2026.

Compression for cross-DC

PowerSGD, 1-bit Adam, and gradient sparsification (top-k) cut communication by 32-1000×. Quality recovers with momentum compensation. Necessary when AllReduce volume is the bottleneck. See decentralized GPU economics for the systems context.

When cross-DC is the right answer

Almost never for a frontier lab with one huge DC. Often for: cooperative open-weight projects (Prime Intellect's Intellect-1), federated medical/finance training, training runs spanning cloud providers for cost.

---

Training reproducibility and bit-exactness

A separate but related concern from determinism in inference.

What's expensive

Bit-identical training across runs (same loss curve to the last decimal) requires: deterministic data loading, deterministic NCCL (NCCL_ALGO=Ring, NCCL_PROTO=Simple, NCCL_NVLS_ENABLE=0), deterministic CUDA kernels (torch.use_deterministic_algorithms(True)), fixed RNG seeds throughout. Performance cost: 20-40%.

What's free

Reproducibility of loss curve at coarse granularity (training to the same downstream evals, not bit-identical) is free if you pin: framework versions, NCCL version, tokenizer version, data ordering seed. Most production "reproducible" training is this kind.

Why frontier labs don't bother

A 90-day training run on 16,000 GPUs has so many sources of non-determinism (hardware failures, network jitter, async optimizer scheduling) that bit-exact reproducibility is unachievable at any cost. Coarse reproducibility — same eval scores within 0.5 points — is the practical goal.

---

Frontier lab training playbooks in 2026

Reconstructed from public statements, papers, and informed inference.

Meta — Llama 3.x and beyond

Stack: Megatron + custom data infra + custom checkpointing. Parallelism: TP=8 × PP=16 × CP=2 × DP=many for 405B. Compute: 16,000 H100s in a single rail-optimized RoCE cluster. Checkpointing: every 1500 steps, ~5 minutes per checkpoint, async to local NVMe then to object store.

Google — Gemini

Stack: JAX + Pathways + XLA. Parallelism: pjit-driven, declared via mesh and sharding annotations. Compute: TPU v5p pods scaling into the tens of thousands of chips. ICI handles intra-pod; OCS (optical circuit switching) handles inter-pod.

Anthropic — Claude

Stack: undisclosed; PyTorch with custom additions. Compute: mix of AWS Trainium, GCP TPU, on-prem H100/B200. Multi-vendor by necessity. Trains use careful data filtering and constitutional AI techniques.

OpenAI — GPT-5 era

Stack: triton-heavy custom kernels, deep CUDA optimization. Microsoft Azure ND-series H100/B200 clusters. Frontier-scale capacity reservations.

DeepSeek — V3, R1

Stack: open-published. Megatron-style TP/EP/PP with DualPipe scheduling — overlaps EP all-to-all with compute. FP8 throughout, including KV cache. 2048 H800s for the V3 training. Compute cost reported at ~$5.6M for the run; widely viewed as the most cost-efficient frontier training to date.

What to imitate

(1) Pin frameworks and dependencies. (2) Checkpoint frequently. (3) Validate data quality before each major run. (4) Profile and tune for your specific topology. (5) Don't blindly copy frontier configs — your scale doesn't need them.

---

CPU offload and SWAP: when memory really runs out

When even ZeRO-3 / FSDP3 isn't enough — for example, training a frontier-scale model on a cluster with limited GPU count — CPU offload is the next memory-saving option. The pattern: move parameters, gradients, or optimizer state to CPU RAM when not in active use, swap them back to GPU on demand.

DeepSpeed CPU offload

DeepSpeed's zero_optimization.offload_optimizer and offload_param flags enable CPU offload at different granularities. The optimizer state offload is the most common (largest single memory consumer, least bandwidth-sensitive); parameter offload is the most aggressive (cuts GPU memory dramatically but slows training significantly).

Throughput cost: 20-40% slowdown for optimizer offload, 50-80% for parameter offload. Worth it only when the alternative is "training doesn't fit at all."

NVMe offload (ZeRO-Infinity)

A further step: offload to NVMe SSD rather than CPU RAM. The bandwidth is much lower (several GB/s per drive vs hundreds of GB/s for HBM), so the throughput hit is severe. The use case is training models too large to fit even in aggregate CPU RAM.

Asynchronous offload

A refinement: prefetch the next layer's parameters from CPU/NVMe to GPU while the current layer is computing. Overlaps the slow transfer with compute, reducing the effective throughput hit. Implementation is non-trivial; DeepSpeed and some other frameworks support it.

When offload is the right answer

For most production training, offload is the wrong answer — buying more GPUs is cheaper than the throughput cost of offload. The exceptions: research workloads, models near the boundary of feasibility, or single-node fine-tuning of very large models on limited hardware.

For more context on the memory hierarchy this builds on, see KV cache inference memory math (the inference side has similar trade-offs).

---

LocalSGD and async data-parallel variants

LocalSGD is a precursor and sibling of DiLoCo: run multiple local SGD steps per all-reduce rather than one. The trade-off is similar — less communication, slower convergence — but the use case is different. LocalSGD targets bandwidth-limited single-cluster training (e.g., training over slow intra-cluster links); DiLoCo targets cross-cluster training.

LocalSGD vs DiLoCo

LocalSGD performs simple averaging of weights across workers every N local steps. DiLoCo adds an outer optimizer (typically Nesterov momentum at the outer loop) that improves convergence. DiLoCo is essentially "LocalSGD with a smarter outer aggregator."

Asynchronous variants

Async-DP variants relax the synchronous requirement entirely — workers update a shared parameter server without waiting for each other. Faster wall-clock, slower convergence due to stale gradients. Used in older parameter-server architectures; mostly displaced by FSDP/ZeRO in modern training.

When async helps

In cluster designs with very heterogeneous hardware (some workers faster than others), async can give throughput gains that synchronous training cannot match. The convergence cost is real but workload-dependent. Not a default choice but a useful tool in specific situations.

---

Frontier training: 2026 case studies

A few public training recipes that codify what the frontier looks like in mid-2026.

Llama 3.3 70B

Meta's Llama 3.3 70B (released late 2024) was trained on a 16K-GPU H100 cluster over roughly 7M GPU-hours. The published recipe used BF16 precision (not FP8), Megatron-style 4D parallelism (DP + TP + PP + SP), and a sequence length of 8K throughout pretraining with long-context fine-tuning afterward. The MFU was reported around 40%, considered very good at that cluster scale.

Llama 4 multimodal

Meta's multimodal Llama 4 family extended the recipe to include image and video inputs alongside text. The training stack added vision tower training (separate ViT-style encoders) and joint multimodal pretraining. Multimodal training adds memory pressure because vision tokens are typically high-resolution; the parallelism plan had to allocate more memory to activation storage.

DeepSeek-V3 (671B MoE)

DeepSeek-V3 (released December 2024) is the public reference for cost-efficient frontier MoE training. The recipe: 671B total parameters, 37B active per token, trained on 14.8T tokens over 2.78M H800-hours. Precision was FP8 throughout with block-wise scaling. Parallelism plan: TP + PP + EP + standard data parallelism, with DualPipe (a custom pipeline schedule that overlaps EP all-to-all with compute).

The reported cost of $5.6M for the training run made it the most cost-efficient frontier training publicly documented. The cost discipline came from a combination of FP8, MoE sparsity, careful curriculum, and aggressive engineering optimization.

DeepSeek-R1 (reasoning model)

R1's post-training pipeline (a GRPO-based RL stage on top of V3-Base) added reasoning capability. The training compute for R1's RL stage was smaller than V3's pretraining — typical RLVR runs are 5-20% of pretraining compute. For deeper analysis of post-training recipes, see post-training RLHF DPO.

Qwen 3 family

Alibaba's Qwen 3 family (mid-2025) spans 1B to 235B parameters with both dense and MoE variants. The published recipes use BF16 with optional FP8, FSDP-style data parallelism, and careful curriculum learning. The smaller Qwen models are particularly notable for matching much larger models on common benchmarks — a result attributed to post-training discipline more than pretraining scale.

Llama 5 (rumored, late 2026)

Public information is partial. Reports suggest a 600B+ MoE design with FP8 throughout, B200 hardware, and 50K-GPU+ cluster scale. Wall-clock time and cost are not yet public.

---

Activation checkpointing in detail

Activation checkpointing (also called gradient checkpointing or recomputation) trades extra compute for less memory by discarding activations during forward and recomputing them during backward.

How it works

Without activation checkpointing, the forward pass stores every intermediate activation needed for the backward pass. For a deep transformer, the activation memory exceeds the model weight memory by 5-10x.

With activation checkpointing, the forward pass stores only activations at certain layer boundaries (the "checkpoints"). During backward pass, the activations between checkpoints are recomputed from the saved boundary state. Memory drops dramatically; compute increases by roughly 33% (one extra forward pass during backward).

Full vs selective checkpointing

Full activation checkpointing recomputes every layer. Maximum memory savings, maximum compute overhead. Selective activation checkpointing recomputes some layers and stores others — the choice is typically based on each layer's memory-vs-compute ratio.

The 2026 best practice: selective activation checkpointing with manual tuning per architecture. The PyTorch checkpoint_wrapper API supports this; Megatron-LM has explicit selective checkpointing built in.

Async checkpointing

A refinement: recompute activations asynchronously while the previous layer's backward is still running. The recompute overlaps with the backward compute, reducing the effective overhead. Implementation is non-trivial; production stacks (Megatron, recent FSDP2) support it.

Memory and compute trade-off

For a 70B model with 80 transformer layers, full activation checkpointing saves around 500GB of activation memory at the cost of 33% additional compute. The trade is almost always worthwhile — without checkpointing, the model doesn't fit at frontier scale.

Selective activation checkpointing typically recovers 10-20% of the compute overhead while keeping most of the memory savings. The win depends on the architecture's specific memory-vs-compute ratios.

When activation checkpointing is the wrong answer

If memory is not the bottleneck (e.g., training a small model on a large cluster), activation checkpointing wastes compute. Disable it in those cases.

If the activation memory is not the dominant memory consumer (e.g., optimizer state is much larger), activation checkpointing helps less. Address optimizer state first via ZeRO-2 or ZeRO-3.

---

Cluster utilization metrics: MFU, HFU, MBU

How do you measure whether your training run is well-tuned? Several metrics, each capturing a different aspect of utilization.

Model FLOPs Utilization (MFU)

The fraction of theoretical peak FLOPs the model is actually achieving. Computed as (actual FLOPs per second) / (peak hardware FLOPs per second). For H100 in BF16, peak is around 1 PFLOP/s per GPU; achieved MFU on a tuned training run is typically 40-55%.

MFU above 50% is excellent for transformer training; 40-50% is typical; below 30% suggests the run is communication-bound or has other inefficiencies.

Hardware FLOPs Utilization (HFU)

Similar to MFU but counts all FLOPs including recomputation from activation checkpointing. HFU is always higher than MFU because activation checkpointing inflates the FLOPs count by roughly 33%.

The relationship: HFU = MFU * (1 + recomputation_fraction). For full activation checkpointing, HFU is about 1.33x MFU.

Memory Bandwidth Utilization (MBU)

The fraction of theoretical peak HBM bandwidth the workload is achieving. For H100, peak is 3.35 TB/s; achieved MBU on a typical training step is 50-70%.

MBU matters for memory-bound operations (layer norms, softmax, optimizer steps). High MFU but low MBU suggests the workload could benefit from kernel fusion. High MBU but low MFU suggests the workload is memory-bound and needs more arithmetic intensity.

Communication utilization

How much of the wall-clock is spent on communication vs computation. Tracked separately for all-reduce, all-gather, reduce-scatter, all-to-all. A well-tuned training run hides most communication behind computation; communication-bound runs have 20-40% of time in raw communication.

How to improve each metric

• Low MFU: check for un-fused operations, suboptimal attention kernels, debug-mode overhead.
• Low MBU: kernel fusion, larger batch sizes, FP8 (which reduces memory pressure).
• High communication time: topology tuning (NCCL_IB_QPS_PER_CONNECTION, NCCL_IB_GID_INDEX), better parallelism plan, gradient accumulation.

Reference numbers for frontier training

Cluster scale	Typical MFU	Typical communication share
64 GPUs	50-55%	5-10%
256 GPUs	45-50%	10-15%
1024 GPUs	40-45%	15-20%
4096 GPUs	35-40%	20-25%
16384 GPUs	30-35%	25-35%

The pattern: MFU degrades and communication share grows as the cluster scales. Frontier labs spend enormous engineering effort to push these numbers in the right direction.

---

Extended FAQ

Q: How do I choose between PP and FSDP for a model that won't fit in TP=8?

PP wins on memory at the cost of pipeline bubbles. FSDP wins on overlap at the cost of more communication. Practical heuristic: use PP if you're already multi-node (network is slow, comm cost is fixed regardless); use FSDP if you're staying within a fast IB fabric where the extra comm is cheap. Modern frontier training uses both — PP for the largest dimension, FSDP within each PP stage.

Q: What's the practical limit on TP within a node?

8 for H100/H200 (NVSwitch). 72 for GB200 NVL72 (rack-scale NVLink). Going beyond NVLink (TP across IB) is almost never worth it — the per-layer AllReduce latency on IB exceeds 1 ms, multiplied by 80 layers per forward pass, ruins step time. The rare exception is very small models where the compute is so fast that even IB-bound TP keeps the GPU busy.

Q: When should I add CP (context parallelism)?

When per-GPU activation memory exceeds available HBM with all other tricks applied. Empirically: CP=2 becomes useful around 32K context with 70B+ models; CP=4 for 64K+; CP=8 for 128K+. Below 32K context, selective recompute + SP within TP handles activations.

Q: Why does my FSDP run slow down at large DP scale?

FSDP issues 2 × num_layers collectives per step. At DP=512, each collective spans 512 ranks, each with params_per_layer / 512 data — small messages dominate. Network overhead per collective scales as log(N), total overhead grows. Fixes: hybrid sharding (HSDP — full-shard within node, replicate across nodes), bigger transformer blocks (FSDPWrapper at deeper levels), bucket_cap_mb tuning.

Q: What's the practical batch size limit for stable training?

Empirically, gradient noise scale (Smith et al., McCandlish et al.) caps useful global batch size at ~`64 × num_active_params × 1e-9`. For 70B dense: ~4.5M tokens. For 405B dense: ~26M tokens. Beyond this, batch size keeps increasing wall-clock but quality plateaus or regresses. Modern frontier training is at or just below this limit.

Q: How do I handle distributed checkpointing for 100B+ models?

Use PyTorch's torch.distributed.checkpoint (DCP) or NeMo's distributed checkpoint format. Each rank writes its shard in parallel; total write time is bounded by per-rank NVMe bandwidth (~3 GB/s) divided by checkpoint size per rank. For 405B FP32 optimizer state at TP=8/PP=16/DP=64, per-rank state is ~1 GB — checkpoint completes in a few seconds. The bottleneck is usually the object store upload, not the per-rank write.

Q: What's the impact of using torch.compile on collective behavior?

torch.compile reorders operations across collective boundaries when it can prove no aliasing. The good news: better compute-collective overlap. The bad news: compile errors can be obscure when collectives are involved. Always benchmark a few hundred steps with and without torch.compile before committing to it for a multi-week run.

Q: How do I handle stragglers without elastic training?

Three options: (1) Tight monitoring — kill the job and restart from checkpoint when one rank is consistently slow. (2) Per-rank step-time logging and operator-level "kick the slow node" runbooks. (3) Use NCCL_TIMEOUT to bound the slowest collective; rank-skipping protocols exist but are complex. For most non-frontier training, option 1 is sufficient.

Q: What's the right ratio of optimizer state offload to HBM?

ZeRO-Infinity offloads optimizer state to CPU memory or NVMe. Cost: extra PCIe traffic per step. Practical use: only when HBM is the binding constraint and you cannot add more GPUs. The optimizer step becomes 2-5× slower; usually not worth it. Better solutions: increase TP/PP, switch to a lower-bit optimizer (8-bit Adam from bitsandbytes).

Q: How do I diagnose NCCL hangs in FSDP training?

Set NCCL_DEBUG=INFO, TORCH_NCCL_BLOCKING_WAIT=0, TORCH_NCCL_ASYNC_ERROR_HANDLING=1, NCCL_TIMEOUT=600. When a hang occurs, NCCL prints stack traces from every rank. The hung collective is named — typically one rank's _allgather_base or _reduce_scatter_base. Cross-reference with FSDP layer numbers to identify which transformer block. Common root cause: a non-deterministic data loader producing differently-sized batches on different ranks.

Q: Should I use BF16 or FP16 for activations?

BF16 always. FP16's 5-bit exponent overflows in attention softmax denominators at long context (>4K) and in some MLP intermediate activations. BF16's 8-bit exponent matches FP32's dynamic range, eliminating the overflow class. The only reason FP16 still exists in 2026 is older GPUs (V100, T4) without BF16 support.

Q: When does FP8 training pay off?

For 70B+ training on H100/H200/B200, FP8 forward + BF16 master gradients gives ~1.7-1.9× speedup with no measurable quality loss (after careful per-tensor scaling). For <13B models, FP8 setup overhead can be larger than its benefit. Always measure on your specific model and data; FP8 is workload-sensitive.

Q: What's the right LR schedule for very large pre-training?

Warmup over the first 1-2% of steps (Llama-3 used 0.5%), constant or cosine decay to ~10% of peak over remaining steps. Some frontier runs use WSD (warmup-stable-decay) — constant LR for most of training, then linear decay at end. WSD gives better predictability and easier checkpoint resumption.

Q: How do I detect training data contamination of evals?

(1) Compute n-gram overlap between training shards and eval sets at decontamination time. (2) Track perplexity on eval sets per training shard — sudden drops suggest contamination. (3) Hold out a private eval set that's never been on the internet. (4) Compare to base model behavior — if the fine-tuned model gets weirdly high scores on a specific benchmark, it might have seen it.

Q: What's the right number of training tokens for a model of size N?

Chinchilla-optimal is ~20 tokens per parameter, but most production training in 2026 uses 100-300 tokens per parameter for downstream task quality. Llama-3 70B saw 15 T tokens — over 200× parameters. Quality scales with tokens past Chinchilla; compute-optimal isn't quality-optimal.

Q: How do I handle vocab size changes between pre-training and fine-tuning?

Don't. Pin the tokenizer at pre-training and never change it. If you must add tokens (e.g., function-call tokens for tool use), initialize them as the average of existing embeddings and freeze them for the first few hundred steps to let surrounding context adapt.

Q: What's the difference between expert parallelism and data parallelism for MoE?

EP shards experts across GPUs — each GPU owns a subset of experts and processes tokens routed to those experts via all-to-all. DP for MoE replicates the entire expert pool — every GPU has every expert, tokens stay local. EP scales to many experts; DP scales to many tokens. Production frontier MoE uses both: EP within node (NVLink can handle all-to-all), DP across nodes.

Q: How do I shard a custom transformer architecture for FSDP?

Define a ModuleWrapPolicy that wraps each transformer block (or each layer if blocks are huge). FSDP shards within each wrapped unit. The grain: too fine wastes communication; too coarse blocks compute. Standard practice: wrap each TransformerBlock — gives good balance of memory and overlap for most architectures.

Q: What's gradient_as_bucket_view in PyTorch DDP?

A DDP flag that makes gradients direct views into the bucket memory, avoiding a copy after backward. Saves ~5% of step time and ~1 GB per replica. Always enable in production: DDP(model, gradient_as_bucket_view=True).

Q: How do I prevent loss spikes during long training?

(1) Gradient clipping at ||g|| ≤ 1.0. (2) Loss scaling (FP16 only) at conservative initial value (2^15). (3) Skip optimizer step when any gradient is NaN/inf. (4) Detect spike (loss > recent_mean + 3 × recent_std), automatic rollback to last checkpoint. (5) Z-loss regularization (penalize logit drift) for very long context.

Q: When should I use Lion or Sophia instead of AdamW?

Lion (Google, 2023) saves ~50% of optimizer memory (no second moment) and trains ~1.2× faster at similar quality. Sophia (Liu et al., 2023) uses Hessian information for ~2× convergence speed. Both are reasonable for new training runs; AdamW remains the default because it's most-tested. For 100B+ scale, the optimizer-memory savings from Lion are meaningful enough to consider seriously.

Q: What's the trade-off between micro-batch size and pipeline bubble?

Larger micro-batches = fewer micro-batches per step = larger pipeline bubble (fewer steps to fill the pipeline). Smaller micro-batches = more micro-batches = smaller bubble but more per-micro-batch overhead. Megatron's interleaved 1F1B with 4-8 chunks per stage and 64+ micro-batches typically achieves <5% bubble for 16-stage pipelines.

Q: How do I migrate from FSDP-1 to FSDP-2?

Replace FullyShardedDataParallel(module) with fully_shard(module) from torch.distributed._composable. State dict handling changes — FSDP-2 uses DTensor natively, so checkpoints are different. Test resharding from FSDP-1 checkpoints with the migration utility before committing. Performance: FSDP-2 is 5-15% faster on most workloads and significantly easier to combine with TP.

Q: What's the right approach for fine-tuning at a single GPU but with a too-big model?

QLoRA: 4-bit quantize the base model, train LoRA adapters in BF16. Lets 70B models fit on a single 80 GB H100. Quality is 95-99% of full fine-tuning. The catch: training is slow (3-5× slower than equivalent BF16 LoRA due to dequantization overhead in forward). For research and small datasets, QLoRA is fine. For production fine-tuning at scale, get more GPUs.

Q: How does Megatron handle very long context training?

Megatron uses Context Parallelism (CP) with ring-attention-like KV exchange between CP ranks. Activations of length S/CP are stored per rank; KV is rotated around the ring during attention. Memory per rank for activations scales as S/CP instead of S. Communication scales linearly with CP × num_layers. Practical: CP=2 doubles max context; CP=4 quadruples but adds 10-20% step time.

Q: What's the impact of EP load imbalance in MoE training?

Experts get unevenly utilized; the over-loaded GPU bottlenecks every step. Auxiliary loss penalizes imbalance during training (DeepSpeed-MoE's aux_loss_weight=0.01). Capacity factor caps tokens per expert and drops excess (DeepSpeed defaults to 1.25). DeepSeek-V3 introduced auxiliary-loss-free balancing — bias terms adjust to enforce balance without quality cost. Worth studying their paper.

Q: How do I plan compute budget for ablations alongside frontier training?

Budget 20-30% of total compute for ablations. Frontier labs spend more on ablation runs than on the headline training run. A 70B reference recipe gets validated on 1B and 7B models at smaller scale first; this is where you find data-mix bugs and hyperparameter issues before they're expensive.

Q: What's the role of FP8 master weights vs BF16 master weights?

FP8 master weights save 50% of weight memory but require careful per-channel scaling. Most 2026 production training keeps BF16 master weights and FP8 forward/backward only — the master-weight memory is a small fraction of total and FP8 master weights still have unresolved stability issues at 100B+ scale.

Q: How do I parallelize evaluation alongside training?

Two approaches: (1) Pause training every N steps, run eval on the training GPUs, resume. Simple but wastes compute during eval. (2) Hold out a few GPUs as a dedicated eval cluster, send checkpoints to them. Better utilization but adds infrastructure complexity. Most teams use option 1 with infrequent (every 5000+ steps) full evals and frequent (every 100 steps) cheap proxy evals (training-loss tracking on held-out shard).

Q: What about training with private data and compliance constraints?

Differential privacy for LLM pre-training is research-grade in 2026. Production "private" training usually means: (1) Data isolation at the cluster level (no internet access, dedicated tenants). (2) Audit trails on data access. (3) Output-side filtering for memorized strings. True DP-SGD at frontier scale destroys quality; nobody does it for general-purpose LLMs.

Q: When should I retrain from scratch versus continue training?

Continue training when: (1) Architecture is unchanged. (2) Tokenizer is identical. (3) Optimizer state is recoverable. (4) Data distribution is similar to original. Retrain from scratch when: (1) Architecture changed (different attention, MoE conversion). (2) Tokenizer changed. (3) Data distribution shifted dramatically. (4) Long enough has passed that recipe-level improvements (better optimizers, FP8) make from-scratch faster.

---