ML Training Reliability: Checkpoints, Fault Tolerance, Recovery, Storage — The Complete Guide

A training run that lasts weeks on thousands of GPUs will fail. Not might. Will. With a node MTBF of, say, ten years, a 10,000-GPU cluster expects a failure roughly every nine hours; a 25,000-GPU cluster expects one every three to four. Nodes drop. InfiniBand links degrade. ECC corrects most single-bit errors but not all of them — and at frontier scale, "rare cosmic-ray strikes" become "one every few days." Jobs get preempted by hardware maintenance, by another team's higher-priority workload, by an upstream networking event. The defense is the checkpoint: a periodic snapshot of training state, written somewhere durable, from which a failed run can resume.

The interesting part isn't that you take checkpoints — that's obvious. It's how much engineering goes into doing it without slowing the training to a crawl, what happens when the recovery path itself has bugs, and how the whole reliability stack — checkpoints, fault tolerance, monitoring, runbooks — comes together so that a 100,000-H100 training cluster doesn't lose a week of progress every time a NIC blinks. Frontier labs treat this as a first-class engineering surface; the difference between a training run that finishes on time and one that slips a quarter is almost entirely upstream of the model architecture.

The take: treat checkpoint and recovery reliability as core engineering, not infrastructure overhead. The cheap shortcuts — naive synchronous writes, no atomic finalization, no checksums, no recovery drills — cost weeks of training time when they fail. The Bamboo and Check-N-Run papers (cited below) document exactly the kinds of failures that hit real production runs, and the fixes are mostly disciplined application of known patterns rather than novel research. The teams that finish ambitious training runs are the ones who invested in this before they needed it. DeepSeek-V3's tech report (arXiv:2412.19437) is unusually candid about how much of training "success" is actually reliability engineering: handling silent data corruption, recovering from straggler nodes, designing the parallelism layout so that failures localize to a small number of ranks.

This guide is about the systems work that keeps multi-week training runs from being multi-week disasters — checkpoints, storage tiers, recovery semantics, fault-tolerance patterns, and the operational practices that hold it all together.

Table of contents

1. Key takeaways
2. Mental model: checkpoint and recovery in one minute
3. The reliability landscape in 2026
4. Failure mode catalog at frontier scale
5. Storage tier economics
6. Async checkpointing with PyTorch DCP
7. DeepSeek / Llama / Anthropic case studies on resilience
8. What's in a checkpoint
9. The two costs: write and recovery
10. Cadence and the lost-work trade-off
11. Storage tiers
12. Synchronous vs asynchronous writes
13. Sharded vs consolidated checkpoints
14. Atomic finalization
15. Recovery semantics
16. Failure modes worth knowing
17. Checkpoint compression and quantization
18. Cross-platform compatibility
19. Production deployments
20. Why this is real engineering investment
21. Elastic training: TorchElastic, Ray Train, and partial recovery
22. Monitoring and runbooks: what to alert on
23. Recovery drills: testing the path before you need it
24. Checkpoint formats: PyTorch, safetensors, DCP, Orbax, GGUF, ONNX
25. FSDP1 vs FSDP2 and ZeRO-1/2/3 checkpoint layouts
26. Storage backends in production: Lustre, WekaFS, VAST, FSx, GCS, S3
27. MTBF math at frontier scale and the cadence equation
28. MoE and LoRA checkpoint specifics
29. Checkpoint provenance and audit (compliance angle)
30. In-memory redundancy patterns
31. Inference-side weight loading
32. Checkpoint compression, deltas, and quantized states
33. Recovery runbook patterns
34. Chaos engineering for training
35. SDC mitigation deep dive
36. Worked example: a 100k-GPU run's checkpoint budget
37. Checkpoint security: encryption, RBAC, and audit
38. Checkpoint versioning: DVC, MLflow, and metadata stores
39. MosaicML Composer and the training-orchestrator angle
40. The Llama 3.1 405B reliability postmortem in detail
41. Cloud-native multipart strategies (S3, GCS, Azure)
42. Async checkpointing libraries: TorchSnapshot, NVIDIA Resiliency
43. Erasure coding vs replication: the cost-curve math
44. Resharding deep dive: how DCP's planner actually works
45. The bottom line
46. FAQ
47. Glossary
48. References

---

Key takeaways

• A training checkpoint includes weights, optimizer state, RNG state, schedule position. Optimizer state is 2-4× the weight size.
• For a frontier-scale training run, checkpoint size is terabytes. Naive synchronous writes pause training for minutes per checkpoint.
• Asynchronous + sharded writes reduce training stall to seconds. Standard in serious infrastructure.
• Cadence rule: checkpoint often enough that expected lost work per failure ≈ checkpoint overhead. Typical: 15-60 minutes.
• Recovery: load latest valid checkpoint, validate completeness, resume. Atomic finalization (write to temp + rename) is mandatory.
• Failure modes: silent corruption, diverged shards, schema drift, storage saturation. All real and expensive.
• A meaningful fraction of total training infrastructure cost goes to checkpoint paths. Treat it as core, not afterthought.

Checkpointing only makes sense in the context of the rest of the training stack: see distributed LLM training for how the parameter, optimizer, and data-parallel shards that you have to serialize get laid out across ranks, and NVLink and rack-scale topology for the intra-rack bandwidth available to the write path.

Quick comparison: checkpoint and recovery strategies

Strategy	Write cost (training stall)	Recovery time	Storage cost	Where it fits
Synchronous full-state	Minutes per checkpoint	10-60 min	1× checkpoint size	Small models, prototyping only
Asynchronous full-state	<30s blocking + bg write	10-60 min	1×	Mid-scale; default in PyTorch DCP
Sharded async (PyTorch DCP, JAX)	Seconds to <10s	Minutes (parallel load)	1× total	Frontier-scale workhorse
Hierarchical / tiered	Seconds (to local NVMe)	Seconds-to-minutes	1-2× across tiers	Frontier production; durability + speed
In-memory replicated (Bamboo-style)	Near-zero blocking	Sub-second to ranks	2-3× across replicas	Preemption-heavy environments
Erasure-coded across ranks	Seconds (encode + write)	Minutes (decode)	1.2-1.5× overhead	Large clusters with cheap intra-rack BW
Lazy / on-demand reconstruction	Negligible	Long (rebuild from logs)	Small	Mostly research; rare in production
Checkpoint-on-failure only	None until failure	Catastrophic	0	Never. Listed for completeness.

The frontier default in 2026 is sharded async + hierarchical storage: each rank writes its shard to a host-RAM staging buffer in seconds, the buffer flushes to local NVMe in the background, and a separate replication process moves snapshots to the cluster filesystem and ultimately to object storage. PyTorch DCP, Megatron-LM's checkpointing, and JAX's Orbax all implement variants of this pattern.

---

Mental model: checkpoint and recovery in one minute

The problem has a name: the recovery tax. At 10k+ GPUs, the cluster fails every few hours; without a fast, valid checkpoint, every minute of training between checkpoints is wasted work, and every minute spent stalled on a synchronous write is also wasted work. The job is to minimize the sum of those two — not to "take checkpoints," but to minimize lost-work plus stall-cost on a curve where naive choices lose weeks of compute over a multi-month run.

The right analogy is save/load in a video game: cheap saves let you take risks; slow saves discourage you from saving at all; corrupt saves are catastrophic. The frontier-scale version adds sharding (parallel writes from every rank), atomic finalization (tmp + rename so a crash mid-write doesn't poison the latest pointer), and tiered storage (RAM → NVMe → cluster FS → object store).

Aspect	Without disciplined checkpointing	With async sharded + atomic
Per-checkpoint stall	Minutes	<2% of step time
Storage write path	Single stream to network FS	Per-rank shard to local NVMe
Finalization	Overwrite-in-place	Write tmp, fsync, rename
Cadence	Hourly (afraid of stall)	Every 15–30 min
Lost work per failure	Up to 1 hour	≤30 min
Silent corruption risk	High (no checksums)	Caught by per-shard hash

The production one-liner is torch.distributed.checkpoint:

import torch.distributed.checkpoint as dcp
# write: async, sharded, atomic
future = dcp.async_save(
    state_dict={"model": model, "optim": optimizer, "rng": rng_state, "step": step},
    storage_writer=dcp.FileSystemWriter(f"{ckpt_dir}/step-{step}.tmp"),
)
future.add_done_callback(lambda _: os.rename(tmp, final))   # atomic finalize
# read: parallel load from all ranks
dcp.load(state_dict=sd, storage_reader=dcp.FileSystemReader(latest_valid))

The sticky number: async sharded checkpointing keeps overhead under 2% of training time at frontier scale (PyTorch DCP, DeepSeek-V3, Llama 3 405B reports). That number is the difference between "we checkpoint every 15 minutes" and "we checkpoint every two hours and pray," and it cascades directly into expected lost work per failure.

---

The reliability landscape in 2026

Training reliability in 2026 is the composition of four sub-problems. Treating any of them as an afterthought breaks the others.

Checkpoint strategies

The mechanical question of how state gets written. Four orthogonal axes:

• Synchronous vs asynchronous — does training pause for the write, or does the write run in the background after a brief GPU-to-host copy?
• Sharded vs consolidated — does each rank write its own piece in parallel, or does a coordinator gather everything into one file?
• Hierarchical vs single-tier — does the checkpoint flow through fast local storage on its way to durable cluster storage, or go directly to the durable layer?
• In-place vs copy-on-write — can the next checkpoint overwrite parts of the previous one, or is each fully independent?

The frontier default is sharded + asynchronous + hierarchical. The other axes get tuned by workload — copy-on-write helps when you keep many recent checkpoints; in-place wins on storage cost when you only keep the last one.

Storage tiers

The state has to live somewhere. The hierarchy: HBM → DRAM → NVMe → cluster parallel filesystem → object store. Bandwidth drops by roughly an order of magnitude at each step; cost-per-byte drops by 2-3 orders by the time you reach object storage. The art is moving data downstream fast enough that the fast tiers stay free for the next checkpoint.

Recovery semantics

When a run fails, what restart mode are you in?

• Warm restart — same processes, same nodes, just reload state and resume. Fastest. Possible if the failure was transient (a watchdog timeout, a recoverable NCCL error).
• Cold restart — re-allocate nodes, re-spawn processes, re-load state. Slow. Required for most hardware failures.
• Partial recovery — most ranks survived; replace only the failed ranks. Possible with elastic frameworks (TorchElastic, Ray Train). Faster than full cold restart but requires careful state reconciliation.
• Replay-from-log — reconstruct lost state by re-running a small number of training steps from the last checkpoint. Used as a finishing step after recovery to ensure determinism.

Fault tolerance patterns

The architectural question: how does the system tolerate failure?

• Plain checkpoint-and-restart — accept the lost work between failure and last checkpoint. The baseline.
• Replicated state — store checkpoint shards on multiple nodes so any single failure has a hot recovery path. Increases storage cost; reduces recovery time.
• Erasure-coded state — store parity blocks instead of full replicas. Lower storage overhead (1.2-1.5× vs 2-3× for replication) at the cost of compute on encode/decode.
• Redundant computation — run important ranks twice and compare outputs. Catches silent corruption that checkpoints alone don't. Expensive; used selectively.

Failure rate math

The number that matters: how often does something fail in a cluster of N GPUs?

Per-GPU MTBF on modern datacenter cards is in the range of 10-50 years for hardware itself, but the system MTBF (driver crashes, NCCL deadlocks, NIC link-down events, host-side OOMs, cosmic-ray-induced ECC errors that the hardware can't correct, silent data corruption that the hardware doesn't even know about) is much lower — often 1-3 years per node. With N nodes, the time-to-first-failure is roughly MTBF/N.

For a 10,000-GPU cluster at 1250 nodes (8 GPUs/node) and a 3-year per-node MTBF: expected first failure every 3 years / 1250 ≈ 21 hours. At 100,000 GPUs: every 2 hours. This is why checkpoint cadence and recovery speed dominate the practical training economics at frontier scale — see Dean & Barroso's "Tail at Scale" (research.google/pubs/the-tail-at-scale/) for the foundational analysis of how failure rates compose in distributed systems.

Cosmic rays, SDC, and link-down events

The unglamorous failures.

• Cosmic-ray-induced single-event upsets — a high-energy particle flips a bit in DRAM or HBM. ECC catches single-bit errors; double-bit errors crash the node; rare correlated multi-bit errors can be silent. Real, measured, and at frontier scale they happen weekly somewhere in the cluster.
• Silent data corruption (SDC) — a chip computes the wrong answer without flagging it. Studied extensively by Facebook/Meta (their 2021 paper on CPU SDC, and Google's similar work) — affects roughly 1 in 1000 chips at some point in their lifetime. For GPUs, SDC manifests as training loss that diverges without warning. Defense: redundant computation on suspect ranks, periodic verification runs, automated outlier detection on per-rank gradient norms.
• NIC link-down events — InfiniBand or Ethernet links flap. The job's all-reduce hangs. NCCL timeouts fire. The job dies. At scale this is the single most common failure mode; see our AI training networking and NCCL guide for the network side.
• Stuck nodes — a rank's training step gets 100x slower than its peers. The straggler holds up every collective. The right response is fast detection + replacement, not patience.

Operational practices

The cron-and-runbook layer.

• Scheduled checkpoint cron — explicit cadence, monitored. If the cron misses, page someone.
• Monitoring dashboards — last-good-checkpoint age, write latency, storage utilization, replication lag.
• Runbooks — exact steps for "recover from a single node failure," "fall back to two-checkpoints-ago," "detect and replace a stuck rank." Written and rehearsed before they're needed.
• Recovery drills — periodic, automated exercises that kill a node mid-training and verify the system recovers. The only way to know the recovery path actually works.

---

Failure mode catalog at frontier scale

A working taxonomy of what actually fails on a 10,000+ GPU training run, organized by frequency and severity. None of this is exotic; all of it has bitten real labs.

Single-node hardware failure (frequent, low severity). A node's NIC dies, a GPU thermal-throttles into a crash, a power supply fails. Detection: heartbeat timeouts, NCCL communication errors. Recovery: replace the node (or fail over to a hot spare), reload checkpoint, resume. With sharded async checkpointing and elastic launch (TorchElastic), this is a 10-30 minute event.

Stragglers (frequent, medium severity if undetected). A rank slows to 10-100x its peers — usually due to thermal throttling, ECC error correction overhead, or a memory-leaky background process. The job runs but at the straggler's pace. Detection: per-rank step-time histograms, outlier alerts. Recovery: kill and replace the slow rank. The cost of missing this is brutal — a single straggler can stall a 10,000-GPU cluster for hours.

NIC link flap / network partition (frequent, high severity). InfiniBand link goes down or partition occurs across a switch. NCCL all-reduce hangs. Without a watchdog, the entire job hangs indefinitely waiting on the collective. Defense: NCCL_TIMEOUT, automated job restart on hang, network-level monitoring.

Silent data corruption (rare, catastrophic). A specific GPU computes wrong outputs. Training loss diverges without obvious cause. Detection: gradient-norm outlier monitoring per rank, periodic redundant computation. Recovery: identify and quarantine the bad GPU, restart from a known-good checkpoint.

ECC double-bit error (rare, high severity). A cosmic-ray strike or memory defect flips two bits that single-error-correction can't handle. The GPU crashes; sometimes the host crashes. Treated as a single-node failure.

Checkpoint corruption (rare, catastrophic). The checkpoint loads cleanly but contains garbage — a torn write, a flipped bit during transfer, a schema-drift bug. Training resumes and diverges. Defense: SHA-256 / BLAKE3 checksums per shard, atomic finalization, multiple-generation retention (so you can fall back two or three checkpoints if needed).

Storage saturation (rare in healthy ops, severe when it happens). Long runs accumulate checkpoints; retention policy isn't enforced; the next write fails or partially fails. Defense: explicit retention cron, monitored utilization, alert before saturation.

Optimizer-state divergence across ranks (rare, subtle). A bug or hardware glitch causes one rank's optimizer state to drift from its peers. Training continues to look healthy on aggregate metrics but slowly degrades. Detection: periodic optimizer-state hash comparison across ranks. Recovery: restart from checkpoint.

Preemption (constant in spot/preemptible environments, otherwise rare). The cluster scheduler reclaims your nodes. The job dies. With Bamboo-style (arXiv:2204.12013) in-memory redundancy across nodes, you can survive most preemptions without losing more than a few seconds of work; without it, you lose everything since the last checkpoint.

Job scheduler / orchestration bug (rare, catastrophic). Kubernetes/Slurm/the bespoke scheduler ships an upgrade; the orchestration layer breaks; running jobs hang or die. Defense: pin scheduler versions for the duration of a run, stage upgrades carefully.

The catalog has two practical uses: it tells you what to monitor (each failure mode → at least one alert), and it tells you what to drill (each failure mode → at least one rehearsed runbook).

---

Storage tier economics

Where you put the checkpoint matters as much as how often you take it. The 2026 hierarchy:

Tier	Typical bandwidth (per node)	Latency	Durability	$/TB-month (approx)	Role in checkpoint flow
GPU HBM	3-8 TB/s	nanoseconds	None (volatile)	n/a	The source
Host DRAM (staging)	~30 GB/s (PCIe 5)	sub-microsecond	None (volatile)	~$3-5/TB-mo amortized	First hop, async write target
Local NVMe	10-20 GB/s	microseconds	Node-local only	~$15-30/TB-mo	Fast durable cache
Cluster parallel FS (Lustre, WekaFS, GPFS, BeeGFS)	100+ GB/s aggregate	milliseconds	Multi-node replication / EC	~$50-150/TB-mo	Workhorse for live checkpoints
Object storage (S3, GCS, Azure Blob)	100 MB/s per stream, vast aggregate	tens of ms	Very high (11+ nines)	~$5-25/TB-mo (with retrieval fees)	Archival, multi-region durability
Cold archival (Glacier, Coldline)	Hours to retrieve	hours	Very high	~$1-5/TB-mo	Long-term retention

What drives the bandwidth budget

A 405B-parameter training checkpoint is ~5.7 TB (weights + optimizer state, Adam + FP32 master). At hourly cadence on 1000 GPUs, that's 137 TB/day of writes. At every-15-minutes cadence, 550 TB/day. A 1000-node cluster filesystem at 100 GB/s aggregate sustained ingest can absorb this comfortably; a 100 GB/s shared bucket cannot.

The hierarchical pattern wins because:

• The GPU → host RAM copy is the only step in the critical path of training.
• Host RAM → local NVMe is fast and decoupled from training.
• Local NVMe → parallel FS is parallel across all nodes; aggregate bandwidth scales with cluster size.
• Parallel FS → object storage is fully asynchronous and uses spare network capacity.

Cost optimization patterns

• Retention pyramid: keep the last 3 checkpoints on local NVMe + parallel FS for fast recovery; keep one per hour for the last 24h on parallel FS; keep one per day for the run duration on object storage; keep one per epoch indefinitely.
• Erasure coding at the parallel FS layer reduces storage overhead from 2-3x (replication) to ~1.2-1.5x with similar durability — at the cost of CPU on read.
• Compression for the object-storage tier (zstd) gives a modest 1.2-1.5x on weight tensors; not worth the CPU cost for the live tier.
• Deduplication across consecutive checkpoints — only some optimizer state changes meaningfully step-to-step. ZeRO-Infinity (arXiv:2104.07857) and related work explore the design space.

For a frontier-scale run, checkpoint storage is typically 1-3% of total infrastructure cost. Cheap insurance.

---

Async checkpointing with PyTorch DCP

PyTorch Distributed Checkpoint (torch.distributed.checkpoint, "DCP") is the open-source reference for sharded async checkpointing in 2026. Worth understanding in detail because it's what most non-frontier teams will actually use (pytorch.org/docs/stable/distributed.checkpoint.html).

The model

DCP separates what you checkpoint (a state-dict) from how it gets persisted (a storage plugin). Each rank contributes its shard of the state-dict; DCP plans the layout, parallelizes the I/O, and finalizes atomically.

import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict

# Save (async)
state_dict = get_state_dict(model, optimizer)
dcp.async_save(state_dict, checkpoint_id=f"step_{step}")

# Load
state_dict = get_state_dict(model, optimizer)
dcp.load(state_dict, checkpoint_id=f"step_{step}")
set_state_dict(model, optimizer, model_state_dict=state_dict[...], ...)

async_save returns a future; training continues. The actual disk write completes in the background. The next async_save call waits for the previous one to finish (or you can manage that explicitly).

What it gets right

• Sharded writes: each rank writes its piece in parallel; aggregate write bandwidth scales with the cluster.
• Topology independence on read: a checkpoint saved on TP=8, DP=64 can be loaded on TP=4, DP=128 (with caveats). DCP's planner reshards as needed.
• Atomic finalization: writes go to a staging area; a single metadata commit promotes the whole checkpoint.
• Pluggable storage: FileSystemWriter for parallel FS, custom writers for S3, GCS, etc.
• FSDP-aware: integrates with torch.distributed.fsdp.FullyShardedDataParallel's state-dict semantics — see FSDP.

What's still tricky

• Host RAM pressure: the async path needs a staging buffer in host RAM sized for the largest single shard's worth of optimizer state. For a 70B model on 8 GPUs, that's tens of GB per node. Plan capacity.
• Optimizer state with custom semantics: any state your optimizer keeps outside the standard optimizer.state_dict() won't be checkpointed. Custom optimizers need a small adapter.
• Schema drift: a code change that renames a parameter breaks load. DCP supports custom load planners that can rename on the fly; you have to write the planner.
• Cross-framework portability: DCP is PyTorch-only. Loading into JAX or TensorFlow requires conversion through a portable format (safetensors).

Tuning knobs

• thread_count on the writer — parallelize I/O within a rank. Useful when writing to a parallel FS that can absorb many concurrent streams.
• single_file_per_rank=False if your storage layer prefers fewer larger files vs many small ones.
• cache_staged_state_dict=True to reuse the staging buffer across consecutive checkpoints. Important for cadences faster than every few minutes.

Megatron-LM's checkpoint utilities, JAX's Orbax, and DeepSpeed's checkpoint manager all implement similar patterns with framework-specific tuning. The DCP model is the most general open-source reference. Combined with mixed-precision training and tensor-parallel layouts from distributed LLM training, it covers most production needs.

---

DeepSeek / Llama / Anthropic case studies on resilience

A few specific examples of what resilience looks like at frontier scale, drawn from published technical reports and engineering blogs. Public details are sparse — these labs share the engineering as a competitive moat — but the broad strokes are visible.

DeepSeek-V3 (December 2024 tech report)

DeepSeek-V3's technical report (arXiv:2412.19437) is unusually candid about reliability engineering. The training run used 2048 H800 GPUs for ~2 months at FP8 mixed precision. Key public details:

• Co-designed checkpoint with parallelism layout — the pipeline-parallel and tensor-parallel partitioning was chosen partly so that single-node failures only affect a localized subset of ranks, minimizing the rebuild surface.
• Frequent checkpoints (every few hundred steps) — the cadence is short enough that any single failure costs minutes of work, not hours.
• Numerical stability monitoring — per-rank loss and gradient norm tracked tightly; outliers trigger investigation rather than being averaged away. Caught a small number of silent compute issues that would otherwise have surfaced as quality regressions weeks later.
• FP8 quantization-aware checkpoint pathways — they had to carefully manage the precision boundaries between checkpoint and live state to avoid drift from re-quantization noise on resume. See mixed-precision training for the FP8 details.

The takeaway: the V3 result is as much an engineering result as a research one, and the engineering work was largely reliability-focused.

Llama 3 (Meta, 2024)

Meta's Llama 3 paper and accompanying engineering posts describe a 16k H100 training run over several months. Public reliability details:

• Per-node failure rate — Meta reported ~one node failure per ~3 hours across the 16k-GPU cluster, in line with the failure-rate math above.
• Recovery time as a tracked SLO — they explicitly measured and optimized "time from failure to resumed training," not just write speed. The end-to-end recovery path matters; optimizing only the write side leaves wins on the table.
• Operational tooling investment — much of the engineering value-add was in monitoring, automated node replacement, and runbooks for common failures.

Anthropic (limited public detail)

Anthropic has shared less about their training infrastructure publicly, but their published responsible-scaling and safety posts mention extensive automated detection of training anomalies and rapid recovery from cluster failures. The implication: the same patterns as the others (sharded async checkpoints, hierarchical storage, automated recovery) at frontier scale.

Common threads

Across these and other publicly described runs:

• Checkpoint cadence is short — 5-30 minutes typical, even at frontier scale.
• Recovery is automated — no human in the loop for routine single-node failures.
• Monitoring is per-rank — aggregate metrics hide stragglers and silent corruption.
• Drills are real — recovery paths are exercised before they're needed.

The labs that finish training runs on schedule are the ones that built this stack before they had a 16,000-GPU cluster, not the ones who scrambled to build it after their first multi-day stall. See distributed LLM training, AI training networking, and NCCL guide for the rest of the training-systems stack that this reliability layer sits on top of.

---

What's in a checkpoint

A complete training checkpoint preserves enough state to resume bit-for-bit from where it left off.

The components

Model parameters. The weights themselves. For a P-parameter model in BF16: 2P bytes.

Optimizer state. For Adam-family optimizers, this is two FP32 buffers per parameter (momentum and variance). With FP32 master weights for mixed-precision training, add another 4P bytes.

Total optimizer + master weights: ~12 bytes per parameter for Adam with FP32 masters. For a 70B model: ~840 GB. For a 405B model: ~4.9 TB.

RNG states. Per-rank RNG state. Without this, resumed runs produce different random samples (data shuffles, dropout masks). Small in size.

Training schedule position. Training step, learning rate, data loader position, any cosine/warmup-schedule state. Tiny.

Metadata. Framework version, model config, hardware topology, intended resumption parameters. Small.

Size budgets

Model size	Weights (BF16)	+ Optimizer (Adam, FP32)	Checkpoint total
7B	14 GB	84 GB	~100 GB
70B	140 GB	840 GB	~1 TB
405B	810 GB	4.9 TB	~5.7 TB
671B (MoE)	1.3 TB	8.1 TB	~9.4 TB

These are the numbers that have to move from GPU HBM to durable storage on each checkpoint. Naive transfer at PCIe bandwidth (16 GB/s) takes minutes.

---

The two costs: write and recovery

Checkpointing has two costs worth keeping separate.

Time to write

While the checkpoint is being saved, training is either paused or contending for IO and network bandwidth — the same fabric that carries the all-reduce and all-gather traffic discussed in AI training networking. Naive synchronous checkpointing on a large model can pause training for minutes.

If you checkpoint every hour and the write pauses for 5 minutes, you're losing 8% of compute time to checkpoint overhead alone.

Time to recover

When a run fails, the time to load the latest checkpoint and reconstitute state determines how much progress is actually lost. A slow recovery turns a 10-minute failure into an hour-long stall.

Recovery time has three components:

• Load state from storage to host memory.
• Distribute shards across GPUs.
• Resume training (warmup back to full throughput).

For terabyte-scale checkpoints on parallel filesystems, recovery is 10-60 minutes. Optimizing it matters as much as optimizing writes.

---

Cadence and the lost-work trade-off

How often to checkpoint? The right answer trades write overhead against expected lost work.

The math

Let c = checkpoint duration (write time, blocking or otherwise). Let T = interval between checkpoints. Let f = expected failure rate (failures per unit time). Let w = expected work between failure and last good checkpoint = T/2 on average.

Total expected lost time per unit:

loss_per_time = (c / T) + f × T/2

Minimize by taking derivative with respect to T and setting to zero:

T_optimal = sqrt(2c / f)

For typical production: c = 10 seconds (with async writes), f = 1 failure per 24 hours = 1/86400 per second.

T_optimal = sqrt(2 × 10 × 86400) ≈ 1300 seconds ≈ 22 minutes

So checkpoint every ~20 minutes for these parameters. In practice, anywhere from 15 minutes to 1 hour.

What changes the answer

• Smaller c (faster checkpoints): can afford more frequent checkpoints.
• Higher f (more failures): more frequent.
• Larger cluster: failure rate scales roughly linearly with GPU count.

A 10,000-GPU cluster has 10× the failure rate of a 1000-GPU one. Checkpointing every 10 minutes may be appropriate at frontier scale.

---

Storage tiers

Where checkpoints actually live. Trade-offs differ by tier.

Local NVMe

• Speed: very fast (10+ GB/s).
• Durability: lost when the node fails.
• Use: first stage of a multi-tier checkpoint. Write locally, copy asynchronously to durable storage.

Cluster parallel filesystem (Lustre, GPFS, BeeGFS, WekaFS)

• Speed: striped across many storage nodes. Aggregate 100+ GB/s.
• Durability: survives single-node failures with replication or erasure coding.
• Cost: dedicated storage hardware. Operationally complex.
• Use: the workhorse for live checkpoints in production training.

Object storage (S3, GCS, Azure Blob)

• Speed: per-stream slow (~100 MB/s); with many parallel streams, can saturate networks.
• Durability: very high; cheap retention.
• Cost: cheap to store, network egress costs for retrieval.
• Use: long-term archival, distributed checkpoints across regions.

Typical tiered architecture

GPU HBM → host RAM → local NVMe → parallel FS → object storage

Each tier is faster than the next but less durable. Checkpoints flow downstream in the background while training continues.

The key practice: write to the fastest available tier first (don't block training waiting for slow tiers), then replicate downstream.

---

Synchronous vs asynchronous writes

The naive checkpoint approach pauses training, writes the entire state, then resumes. For terabyte-scale checkpoints, this pause is unacceptable.

Asynchronous (overlapped) writes

The fix: copy state from GPU HBM to a host-memory staging buffer (fast, ~30 GB/s over PCIe), then resume training. The slow writes (host → storage) happen in the background while training continues.

1. Training pauses briefly.
2. GPU → host RAM copy (the only blocking step).
3. Training resumes.
4. Background: host RAM → storage.

The blocking step is now bounded by PCIe bandwidth, not storage bandwidth. For a 1 TB checkpoint at 30 GB/s: ~33 seconds of stall.

With pipelining (start writing layer L's tensors to host before all layers finish on GPU), this can drop to under 10 seconds.

Risks

• Failure during async write: the in-flight checkpoint is incomplete and unusable. Recovery falls back to the prior checkpoint. Acceptable as long as failure rate is reasonable.
• Host memory pressure: the staging buffer takes serious host RAM. Plan capacity accordingly.
• Network contention: async writes share network with training collectives. Quality-of-service or separate channels help.

In-network checkpointing

Some advanced setups perform the checkpoint write during training collectives, using otherwise-idle bandwidth. Engineering-intensive; mainly seen at frontier scale.

---

Sharded vs consolidated checkpoints

In distributed training, each GPU holds a shard of the model and optimizer state (under FSDP, tensor parallelism, etc.). Two write strategies.

Sharded

Each rank writes its own shard. Parallel writes; very fast.

• Pros: write speed scales with cluster size.
• Cons: checkpoint tied to the specific topology. Resuming requires the same number of GPUs in the same partitioning, or a separate redistribution step.

Consolidated

All shards are gathered onto rank 0 (or a small group) before writing. One coherent file.

• Pros: topology-independent. Useful for archival, publishing, or resuming with different parallelism.
• Cons: slow to write (the gather is bandwidth-heavy). Requires enough memory on the gathering rank.

Practical pattern

• Live training: sharded for speed.
• Archival / handoff: periodic consolidation for portability.
• Tooling: scripts to convert between sharded and consolidated formats.

Many training stacks automate this; you save sharded and periodically run a consolidation job.

---

Atomic finalization

A partial checkpoint is worse than no checkpoint — it can load successfully but produce garbage on resume.

The standard pattern

1. Write to a temporary path: /checkpoints/step_12345.tmp/.
2. Validate completeness: all expected shards present, all checksums match.
3. Atomic rename: mv step_12345.tmp/ step_12345/.

Recovery picks the most recent directory matching the canonical pattern. Any .tmp directory is in-progress or aborted; ignored.

Distributed atomicity

For sharded writes, all shards must finalize together. Each rank writes its shard to a temp location; a coordinator confirms all are present; then a single rename promotes the whole directory.

Without this, recovery might find some new shards and some old, producing inconsistent state.

Validation on read

When loading, verify:

• All expected shards present.
• Checksums match.
• Schema version compatible.
• Metadata describes the same model config.

A checkpoint that fails any of these is treated as invalid; fall back to the prior one.

---

Recovery semantics

When a job fails, the recovery sequence:

1. Detect failure. Job scheduler notices missing heartbeats, failed health checks.
2. Decide to restart. Same nodes? Replacement nodes?
3. Allocate resources. Wait for nodes to be ready.
4. Load checkpoint. Most recent valid one.
5. Resume. Training continues from the checkpointed step.

Important details

Same topology, different node identities. Nodes are interchangeable; you don't need the exact same physical machines. As long as the cluster shape is the same, sharded checkpoints load cleanly.

Different topology. If you must change parallelism (fewer GPUs available, different TP degree), use the consolidation tools to reshard the checkpoint.

Data loader state. Resume from the correct position in training data. Determined by the saved data-loader state.

Wall-clock vs step-clock. After resume, learning-rate schedules and other step-based logic continue based on the step count, not wall-clock. The recovery time is "lost" from a wall-clock perspective but not from a training-progress perspective.

Recovery testing

Test the recovery path in non-production. Pre-production exercises:

• Kill a node, restart, verify it resumes correctly.
• Corrupt a checkpoint, verify the system falls back to an earlier one.
• Verify training metrics match what would have happened without the failure.

Many real failures expose bugs in the recovery path. Better to find them in test.

---

Failure modes worth knowing

Production patterns that bite.

Silent corruption

A bit flips during write or storage. The checkpoint loads cleanly but training produces NaN or diverges.

Defense: checksums (hash on write, verify on read). SHA-256 per shard is cheap and catches everything.

Diverged shards

Different shards from different training steps mixed together. Atomic finalization prevents this; loose discipline breaks it.

Defense: strict atomicity (write to temp, rename only when all shards present). Verify step numbers match across shards on load.

Storage saturation

A long run accumulates checkpoints. Without retention policy, storage fills, next write fails, run stalls (or worse, fails silently).

Defense: explicit retention policy (keep last N checkpoints, plus every Mth for longer history). Monitoring on storage utilization. Alerts before saturation.

Schema drift

Framework version changes the state-dict format. Old checkpoints don't load with new code, or load but with wrong semantics.

Defense: version every checkpoint. Explicit migration tooling for schema changes. Test recovery against old checkpoints periodically.

Stale checkpoint cache

A bug saves checkpoints but the metadata pointer doesn't update. Recovery loads a much older checkpoint than intended.

Defense: monotonic step numbers in checkpoint names. Recovery picks max-step explicitly, not by metadata.

Cross-region replication lag

Multi-region setups: a region fails over to one where replication is lagging. The "latest" checkpoint isn't actually latest.

Defense: replication monitoring; failover decisions consider replication freshness.

---

Cosmic rays and silent data corruption: the math at scale

The unglamorous failure mode that bites every frontier lab eventually. Worth the math because the abstract "rare bit flips" undersells how bad it gets at scale.

Single-event upsets (SEUs) from cosmic rays

Cosmic-ray-induced single-bit errors in DRAM happen at a rate of roughly 1 error per gigabit per ~30 years at sea level (rates vary by altitude, hardware, and ECC effectiveness). For a 100,000-GPU cluster with 96 GB HBM per GPU = ~9.6 PB of memory:

• Total memory: 9.6 × 10^15 bytes = 7.7 × 10^16 bits = 7.7 × 10^7 Gbit.
• Errors per 30 years: ~7.7 × 10^7.
• Errors per day: ~7000.

ECC catches almost all single-bit errors. Double-bit errors, which ECC can't correct, are much rarer (~1/1000 of single-bit), so roughly 7 uncorrected errors per day on a cluster of this size. Each one crashes a node. The cluster's MTBF on this failure mode alone is ~3 hours.

Silent data corruption (SDC)

The Meta CPU-SDC paper (Dixit et al., 2021) reported observable silent corruption in ~1 in 1000 CPUs over their service life. GPU-SDC rates are less well-studied but similar in order of magnitude. For a 10,000-GPU cluster, expect a handful of GPUs to silently produce wrong outputs at some point in their service life. The failure presents as: training loss diverges with no obvious cause, restart from checkpoint produces the same divergence, the bad GPU is identified by per-rank gradient-norm outlier monitoring or by elimination.

Defense in depth

Layer	Catches	Cost
HBM ECC	Single-bit errors	Free (hardware)
Checkpoint checksums (SHA-256 / BLAKE3)	Corruption during transfer/storage	Few % CPU
Per-rank gradient-norm outlier monitoring	SDC in compute	Minimal
Periodic redundant computation on sampled ranks	Validates compute correctness	Modest (~1% extra compute)
Cross-rank state-hash comparison	Optimizer-state divergence	Cheap
Recovery from N-back checkpoint on suspected corruption	Recovers from undetected corruption	Lost work between corruption and detection

Frontier labs run all of these. Below frontier scale, the first three are essential; the rest are optional based on observed failure rates.

Why this matters for checkpoint design

A checkpoint that loaded "cleanly" can still contain corruption if the checksum was wrong or computed against already-corrupt data. The defense: SHA-256 on the write side, verify on read side, and keep multiple generations so you can fall back two or three checkpoints if the most recent is suspect. This is why frontier labs keep 10+ checkpoint generations even when the optimal write cadence suggests a smaller retention pyramid.

For the broader hardware-reliability story including ECC, NIC failures, and link-down events, see AI training networking and NCCL tuning.

---

Checkpoint compression and quantization

Reducing checkpoint size has real benefits — faster writes, less storage, faster recovery.

Compression

Generic compression (zstd, gzip) on weight tensors: typically 1.2-1.5× compression. Modest.

Specialized compression: exploit weight distributions, sparsity patterns. Better ratios but framework-specific.

Quantization

Save weights at lower precision (FP8 instead of BF16): 2× smaller. Optimizer state in BF16 instead of FP32: 2× smaller for those tensors.

Trade-off: precision loss on resume. Most practitioners avoid quantizing the training checkpoint (the master weights need to stay full precision); they quantize only for archival or inference deployment.

Selective checkpointing

Only checkpoint what's necessary to resume. Some intermediate state (gradient accumulators in some setups) can be reconstructed from saved state. Saves space at minor recovery-complexity cost.

---

Bamboo, Check-N-Run, and in-memory redundancy

The Bamboo paper (Thorpe et al., NSDI 2023) made a sharp argument: for preemption-heavy environments (spot instances, shared clusters), disk-based checkpoints are too slow to recover from. The fix is in-memory redundancy: each rank's state is also stored in the host RAM of a neighbor rank. Failure of any single rank is recovered in seconds by pulling state from the neighbor, not from disk.

What Bamboo gets right

• Sub-second recovery from single-rank failures. No disk I/O on the recovery path.
• Tolerant of frequent preemptions. Spot-instance training becomes viable.
• Cheap relative to checkpoint storage. Memory is dedicated to redundancy but the alternative (more frequent checkpoints to durable storage) costs more.

What it costs

• 2-3× host RAM consumption. Each rank holds its own state plus a neighbor's. Tight on memory budgets.
• Engineering complexity. Coordinating in-memory replicas across many ranks is non-trivial; failure detection and re-replication are subtle.
• Doesn't replace durable checkpoints. Bamboo handles single-rank failures; correlated failures (rack down, network partition) still require durable storage.

Check-N-Run

Check-N-Run (Eisenman et al., NSDI 2022) is the recommendation-systems-flavored counterpart. The contribution: differential checkpointing that only writes the changed embedding rows since the last checkpoint, plus tiered storage that keeps recent checkpoints in fast tiers and older ones in cold storage. Reduces write bandwidth by orders of magnitude for recsys workloads where most of the state is sparse embedding tables.

The pattern generalizes beyond recsys: any model with sparse update patterns benefits from differential checkpointing. For dense LLMs, most parameters change every step, so the win is smaller.

When to invest

In-memory redundancy and differential checkpointing are advanced techniques. Most teams should reach for them only after the basic async + sharded + hierarchical pattern is solid and the residual lost-time-from-failures is still unacceptable. At frontier scale (10k+ GPUs, preemption-heavy environments), they pay back. Below that, simpler is better.

Combining with elastic frameworks

Bamboo-style in-memory redundancy pairs naturally with TorchElastic / Ray Train — the elastic framework handles rank-membership changes; in-memory redundancy provides the fast recovery path. The combination gives recovery times of seconds for single-rank failures and minutes for larger failures, vs the tens-of-minutes baseline of disk-only recovery.

---

Cross-platform compatibility

A checkpoint from one framework / topology should ideally load on another.

Within a framework

Generally straightforward. PyTorch DDP / FSDP / DeepSpeed each have their own formats; conversion tools exist.

Across frameworks

Harder. PyTorch ↔ JAX ↔ TensorFlow conversion is possible but lossy in edge cases (state-dict naming, layer numbering, optimizer-state representation).

Across hardware vendors

NVIDIA ↔ AMD: usually fine for weights, sometimes needs conversion for optimizer state (mixed-precision details vary).

Best practice

Save consolidated checkpoints periodically in a portable format. Treat sharded checkpoints as ephemeral and topology-specific.

---

Production deployments

What real systems do.

PyTorch FSDP + sharded checkpointing + async writes + parallel filesystem: the workhorse for many open-source training runs.

Megatron-LM + custom sharding + tiered storage: NVIDIA's flagship training framework, used by many frontier labs.

JAX / Pax + asynchronous distributed checkpointing: Google's stack.

Cloud-native setups: train on AWS / GCP / Azure with EFS / FSx / Filestore for checkpoints, S3 / GCS for archival.

Open-source tooling:

• torch.distributed.checkpoint for distributed PyTorch.
• Megatron-LM's checkpoint utilities.
• safetensors for the file format (replacing pickle for safety reasons).

---

Why this is real engineering investment

A casual reading of training infrastructure makes it sound like checkpoints are a footnote. They aren't.

For a frontier-scale training run:

• Several percent of total infrastructure cost goes to checkpoint storage and IO.
• Engineer-time on recovery debugging is non-trivial — a recovery bug can cost weeks.
• The cadence and discipline of checkpointing directly determines training reliability.

Frontier-lab training infrastructure teams treat checkpoints as a first-class engineering surface. The investment pays off because:

• A working recovery path turns a 1-hour failure into a 5-minute hiccup.
• A broken recovery path turns the same failure into a multi-day rebuild.

The difference between a training run that finishes and one that doesn't is often the quality of the checkpoint system.

For inference, the analogous problem (fast recovery of a serving replica from a known-good model state) is much simpler because the state is read-only. But for training, the discipline around save and restore is one of the quiet competitive advantages of mature infrastructure teams.

---

Elastic training: TorchElastic, Ray Train, and partial recovery

The default checkpoint-and-restart pattern restarts every rank when any one fails. At frontier scale, this is wasteful — losing 8 GPUs out of 10,000 shouldn't require restarting 10,000 processes. Elastic training frameworks let you replace failed ranks while the rest keep running.

TorchElastic

TorchElastic (built into PyTorch as torch.distributed.elastic since 1.9) coordinates rank membership via a rendezvous backend (etcd, c10d, or a managed service). When a rank fails:

1. The remaining ranks detect the failure (heartbeat timeout or NCCL error).
2. They re-rendezvous at the next agreed-upon checkpoint.
3. Failed ranks are replaced; the new rank loads the checkpoint shard for its rank position.
4. Training resumes.

The win: a 30-second failure becomes a 30-second pause, not a 30-minute full restart. The catch: the parallelism layout has to tolerate the membership change. TP=8 groups must still be intact after the swap; a failure that takes out half a TP group requires more sophisticated recovery.

Ray Train

Ray Train (part of the Ray ecosystem) wraps similar functionality with a Ray-actor-based programming model. Each rank is a Ray actor; the cluster manager restarts failed actors. Integrates with Ray's broader fault-tolerance machinery (object store replication, task retry policies).

The trade-off vs TorchElastic: Ray Train is more opinionated and integrated; TorchElastic is more lightweight and PyTorch-native. Frontier labs typically build custom resilience layers on top of either.

What "partial recovery" actually does

A partial recovery loads only the shards belonging to the replaced ranks, leaving live ranks untouched. The math: if 1% of ranks fail, partial recovery is ~100× faster than full restart. The implementation challenge is ensuring the replaced rank's state is bit-identical to what the failed rank would have had at that step — including optimizer state, RNG state, and data-loader position.

Where it breaks

• Across-rack failures that take out an entire TP group. The TP group has to be re-rendezvoused; effectively a small full restart.
• Stale optimizer state if the replacement rank's checkpoint is older than the rest of the run. Use the latest checkpoint and roll back the live ranks to match.
• Diverged state if a Byzantine failure caused one rank's state to drift before failure. Defense: periodic state-hash comparison across ranks.

Practical recommendation

For training runs under 10k GPUs: TorchElastic is sufficient and the integration cost is low. Above 10k GPUs, frontier labs build custom layers that handle multi-rank correlated failures, network-partition events, and faster-than-checkpoint-cadence recovery. The complexity is real; the ROI is correlated with cluster size and run duration.

---

Monitoring and runbooks: what to alert on

A reliable checkpoint system is the one that wakes you up before silently failing, not after. The monitoring layer is what separates "we have checkpoints" from "our checkpoints work."

Required metrics

Metric	Healthy range	Alert threshold
Last-good-checkpoint age	< 1.5× cadence	> 2× cadence
Checkpoint write latency P50	< 30s	> 60s (degradation)
Checkpoint write latency P99	< 2× P50	> 3× P50 (tail)
Checkpoint write success rate	> 99.5%	< 99%
Storage utilization (parallel FS)	< 75%	> 85%
Replication lag to object store	< 1 hour	> 4 hours
Per-rank step-time variance	< 5%	> 15% (straggler)
Per-rank gradient-norm outliers	within 3σ	> 5σ (SDC suspect)
NCCL timeout count	0/hour	> 1/hour

Each metric has a defensive purpose. Last-good-checkpoint age catches checkpoint-write failures that aren't surfaced as errors. Storage utilization catches the slow-burn that ends in a write failure. Per-rank gradient-norm outliers catch silent data corruption before it propagates.

Alert routing

Not every alert is a page. The hierarchy:

• Page: cluster-down, checkpoint-write-failure on the only valid checkpoint, training has stalled for > 10 minutes.
• Ticket: storage filling, replication lag, per-rank stragglers, ECC error rate increasing.
• Dashboard: utilization trends, per-checkpoint timing, recovery-drill success rates.

Underreacting (everything is a page) burns out the on-call; overreacting (everything is a dashboard) misses real failures.

Runbook discipline

Every alert needs a runbook entry. The minimum content:

1. What the alert means in plain language.
2. The first three things to check (often a dashboard query, a log search, a recent change).
3. The known fixes ranked by probability.
4. Who to escalate to and when.

Runbooks rot — code changes, infrastructure changes, the runbook still references the old way. Quarterly review of runbook accuracy is the discipline that keeps the on-call effective.

---

Recovery drills: testing the path before you need it

The most common reason training-recovery fails in production: the recovery path was never tested under realistic conditions. The path exists in code, runs once during initial bring-up, then sits unexercised until needed — and by then, code changes have broken it.

What to drill

• Single-node failure mid-step. Kill a worker, verify rendezvous and recovery.
• Network partition. Drop a switch's connectivity briefly, verify NCCL timeout and recovery.
• Slow node. Inject artificial latency into one rank's collective, verify straggler detection.
• Checkpoint corruption. Truncate the latest checkpoint, verify fallback to the prior one.
• Storage saturation. Fill the parallel FS to 100%, verify graceful degradation rather than silent corruption.
• CDU failure simulation. On rack-scale hardware, simulate cooling-system trip; verify thermal throttling behavior and job restart.

Drill cadence

The frontier-lab pattern: monthly automated drills exercising the most common failure modes, quarterly tabletop exercises with the on-call team walking through novel scenarios, semi-annually full chaos-engineering days where multiple correlated failures are injected. Below frontier scale, monthly automation is often enough.

What drills reveal

In our experience, the typical drill catches:

• One or two stale runbook references.
• A monitoring blindspot (the alert that "should have" fired didn't).
• An unexpected dependency (the metadata store that has to be available for recovery to work).
• Slow recovery times that wouldn't have been noticed without measurement.

Without drills, these accumulate silently. With drills, they're surfaced and fixed before the real failure forces it.

Drills as onboarding

A new ML-infra engineer's first month should include running a recovery drill end-to-end. It's the fastest way to internalize how the failure-and-recovery path actually works — far better than reading documentation. The companion guides distributed LLM training, NCCL tuning, and AI training networking become much more concrete after one drill cycle.

---

Checkpoint formats

The format choices proliferated through 2024–2025 and consolidated in 2026. Each serves a different operational concern.

PyTorch native (.pt, .pth)

Python pickle under the hood. Loads anything Python can serialize — including arbitrary code, which is the well-known security caveat. Still common for research and small-model handoffs because it Just Works inside PyTorch. Avoid for any checkpoint you'll share, publish, or load on untrusted infrastructure.

safetensors

The Hugging Face replacement for pickle (github.com/huggingface/safetensors). Memory-mapped, zero-copy, type-safe, language-agnostic. The header is a JSON blob describing dtypes, shapes, and byte offsets; the payload is raw tensor bytes. No arbitrary-code-execution surface. By 2026 the de-facto format for published model weights — every frontier release on Hugging Face ships safetensors. Doesn't carry optimizer state directly; that's a separate consideration.

PyTorch Distributed Checkpoint (DCP)

The sharded, parallel, framework-native format. State dictionary fans out across ranks into a directory of .distcp shards plus a .metadata file describing the global tensor-to-shard mapping. Reshard-on-load is a first-class operation — load a checkpoint written under TP=8 PP=4 into a TP=4 PP=8 layout. The 2026 default for serious PyTorch training. See pytorch.org/docs/stable/distributed.checkpoint.html.

GGUF

llama.cpp's container format for quantized weights, designed for inference on consumer hardware. Includes the model architecture metadata and quantization scheme alongside the weights. Not used for training checkpoints; the dominant format for inference-only deployments of open-weight models. Mentioned because it's the format your shipped model often ends up in after a final conversion step.

ONNX

The cross-framework graph format. Useful for inference deployment across runtimes (ONNX Runtime, TensorRT, OpenVINO). Less useful as a training-checkpoint format — limited support for the training-only state (optimizer momentum/variance) — and slower to evolve with new model architectures. Mostly an export target, not a checkpoint format.

JAX Orbax / Flax

Orbax is the JAX-side equivalent of PyTorch DCP — sharded async checkpoints with reshard-on-load, integrated with jax.distributed. Flax's older flax.training.checkpoints API is being deprecated in favor of Orbax. Async-by-default on TPU pods, where the host RAM and the slice's high-bandwidth interconnect make the staging-buffer approach especially fast.

DeepSpeed and Megatron-LM formats

Framework-specific binary layouts that match each system's parallelism model. DeepSpeed's checkpoint is per-rank .pt files plus a zero_to_fp32.py converter that consolidates ZeRO-3 shards back to a flat model. Megatron-LM uses a tensor-parallel sharded layout with its own metadata. Both can interop with Hugging Face checkpoint formats through conversion scripts; the conversion is non-trivial and a common source of bugs at handoff time.

NeMo Framework

NVIDIA's framework wraps Megatron-LM with additional checkpoint utilities. By 2026 NeMo's checkpoint format is converging on a "distributed checkpoint" pattern compatible with DCP semantics. Used heavily inside NVIDIA-shop training stacks.

Hugging Face Accelerate

Wraps the underlying framework's checkpointing (DCP for FSDP, DeepSpeed for DS) with a unified API. Useful when a training script wants to switch between FSDP and DeepSpeed without rewriting the checkpoint code. Not a format itself — it delegates to whichever framework is active.

Comparison table

Format	Sharded	Reshard-on-load	Carries optimizer state	Security	2026 fit
.pt pickle	No	No	Yes	Unsafe to load untrusted	Research only
safetensors	No	No	No (weights only)	Safe	Published weights
PyTorch DCP	Yes	Yes	Yes	Safe	PyTorch training default
Orbax	Yes	Yes	Yes	Safe	JAX training default
DeepSpeed	Yes	Via converter	Yes	Safe	ZeRO-based training
Megatron-LM	Yes	Limited	Yes	Safe	NVIDIA-shop training
GGUF	No	No	No	Safe	llama.cpp inference
ONNX	No	No	Limited	Safe	Cross-runtime export

---

FSDP1 vs FSDP2 and ZeRO-1/2/3 checkpoint layouts

The two dominant sharded-training systems each ship multiple checkpoint generations; understanding the differences saves you a migration disaster.

FSDP1 vs FSDP2

FSDP1 (PyTorch's original Fully Sharded Data Parallel) flattens per-layer parameters into a FlatParameter and shards that. The checkpoint format reflects this: shards are slices of flattened buffers, and reconstructing the per-layer structure on load requires the same wrapping policy that was used at save time. Hard to interoperate across model architectures.

FSDP2 (released 2024) shards via per-parameter DTensor instead, removing the FlatParameter abstraction. Checkpoint format is now per-parameter sharded; DCP handles it natively; reshard-on-load works across different DP/TP/PP layouts without manual fiddling. Migration note: FSDP1 checkpoints are not directly loadable in FSDP2 — you need to either convert via a consolidated intermediate or maintain parallel save paths during the migration. The PyTorch team published migration guides; budget engineer-weeks for the cutover.

ZeRO-1, ZeRO-2, ZeRO-3

• ZeRO-1 shards optimizer state across DP ranks only. Each rank holds full weights and gradients, only its slice of optimizer state. Checkpoint: each rank saves its optimizer slice; weights gathered to rank 0 (or all ranks).
• ZeRO-2 adds gradient sharding. Each rank holds full weights, partitioned gradients, partitioned optimizer state. Checkpoint adds the gradient shards (though gradients are usually transient and not checkpointed).
• ZeRO-3 shards weights too — each rank holds only a slice of weights at rest, gathering them just-in-time during forward and backward. Checkpoint: per-rank weight slices, optimizer slices, possibly gradient slices.

DeepSpeed's zero_to_fp32.py script consolidates ZeRO-3 shards into a single flat model checkpoint suitable for inference handoff or interoperability. The script is slow on large models (sequential reads from each shard); for frontier-scale models the consolidation step alone can take hours.

Async checkpointing in FSDP2

FSDP2 + DCP support async checkpointing where the per-rank shard is copied to a host-RAM staging buffer in the foreground (seconds), and the background process drains the buffer to durable storage. The training stall is bounded by the staging copy time, not the durable write. NVMe in the compute nodes acts as a third tier: stage to RAM, flush to local NVMe (still tens of seconds), replicate to cluster FS / object store in background.

Comparison

System	Shards weights	Shards optimizer	Async support	Reshard on load
FSDP1	Yes (FlatParameter)	Yes	Limited	No
FSDP2	Yes (DTensor)	Yes	Yes (with DCP)	Yes
ZeRO-1	No	Yes	Yes	Limited
ZeRO-2	No (grads sharded)	Yes	Yes	Limited
ZeRO-3	Yes	Yes	Yes	Via converter

---

Storage backends in production

The parallel filesystem under your training cluster determines your checkpoint throughput ceiling.

Lustre

The HPC workhorse. Open-source, mature, scales to exabytes. Used by most national supercomputing centers and several frontier AI labs. Throughput depends entirely on the number of OSTs (object storage targets) and the per-OST bandwidth — a well-provisioned Lustre filesystem can sustain 1+ TB/s aggregate writes. Weakness: operational complexity is high; tuning is a specialty.

BeeGFS

Open-source parallel filesystem, lighter operational burden than Lustre, common in mid-scale clusters. Per-cluster throughput typically 100–500 GB/s.

WekaFS

Commercial parallel filesystem optimized for low-latency small-file workloads and high-bandwidth large-file workloads. Common in AI cloud providers and at enterprises that don't want Lustre's ops burden. Throughput scales with NIC count; 200–1000 GB/s achievable.

VAST Data

Commercial, all-flash, scales to multi-petabytes with single-tier semantics. Used by several AI labs for the combination of training-checkpoint throughput and serving-tier latency. Higher cost per TB than HDD-based tiers; lower cost per IOPS.

DDN (EXAScaler, Infinia)

Commercial HPC storage vendor, common in NVIDIA-shop training clusters. EXAScaler is Lustre-based; Infinia is DDN's newer all-flash platform aimed at AI. Throughput numbers similar to Lustre/WekaFS at the high end.

AWS FSx for Lustre, Azure NetApp Files, GCS

Cloud-managed parallel filesystems. FSx for Lustre is the AWS go-to for training-checkpoint paths; throughput scales with provisioned capacity (1.2 GB/s per TB at the high tier). Azure has multiple options (NetApp, Managed Lustre, Blob NFS); GCP offers Filestore and the newer Parallelstore. All-tier costs are higher than self-hosted, but operational savings often justify it for non-frontier scale.

Object stores (S3, GCS, Azure Blob)

Cheap, durable, slow per stream. The dominant pattern: use object storage as the durable archive tier; never as the primary checkpoint write target during training. Multipart uploads parallelize the write (8–16 MB part size, dozens of concurrent parts) so aggregate throughput is good even though per-stream is modest. Restore-time parallel reads work well too.

NVMe in compute nodes

Modern training nodes (H100/H200/B200 servers) ship with multiple TB of local NVMe per node. The 2026 pattern: stage checkpoints to local NVMe in seconds; replicate to the cluster filesystem and object store in background. Local NVMe is the fastest tier (10+ GB/s per drive); the cluster FS is durable; object store is archival. A node failure loses its local NVMe shard, but the replicated copy on the cluster FS is still good.

Per-NIC throughput

The fundamental bandwidth limit is per-NIC. A 200 Gb/s NIC delivers ~25 GB/s before protocol overhead, ~20 GB/s sustained. A node with two 400 Gb/s NICs can push ~100 GB/s if the storage backend keeps up. At cluster scale, per-NIC bandwidth × node count is the upper bound on checkpoint write throughput. A 4,096-node cluster with 200 GB/s per node has 800 TB/s aggregate; in practice the storage backend caps it lower. Plan for 30–50% of theoretical as the realistic sustained number.

Comparison

Backend	Type	Throughput at scale	Cost shape	Best for
Lustre	Self-hosted HPC	1+ TB/s	High opex	National-lab and frontier scale
BeeGFS	Self-hosted	100–500 GB/s	Medium opex	Mid-scale
WekaFS	Commercial	200–1000 GB/s	Higher capex	Enterprise / cloud
VAST	Commercial all-flash	1+ TB/s	High $/TB	AI labs
DDN	Commercial HPC	1+ TB/s	High opex	NVIDIA-shop training
AWS FSx for Lustre	Managed	Up to ~1 TB/s provisioned	Cloud rate	AWS training
GCS / S3	Object	Per-stream low; aggregate high	Cheap	Archive only
Local NVMe	Per-node	10+ GB/s per drive	Built into node	Staging tier

---

MTBF math at frontier scale

The cadence question reduces to a single equation once you measure failure rate.

Per-GPU MTBF

Field data from large clusters (Meta's 16k H100 cluster paper, the Falcon-180B training postmortems, the OPT-175B logbook) suggests per-GPU MTBF in the range of 2–8 years for catastrophic failures requiring restart. Call it 5 years as a working number. A 10,000-GPU cluster expects a failure every 5 years / 10,000 ≈ 4.4 hours. A 100,000-GPU cluster expects one every 26 minutes.

Add in network failures (NIC drops, switch failures, cable issues), storage failures, and software bugs, and the effective MTBF — the rate at which the job actually has to recover — is typically 2–4× lower than the GPU-only number. So a 25,000-GPU cluster realistically sees a recoverable failure every 30–90 minutes.

Cadence equation

For a checkpoint cadence T and per-checkpoint write time W, expected lost work per failure is T/2 + recovery_time. Expected overhead per hour is W/T (fraction of training time spent checkpointing) + (failure_rate × (T/2 + recovery_time)). Differentiating with respect to T gives the optimum cadence: T* ≈ sqrt(2 × W / failure_rate).

Worked example: W = 30 seconds per checkpoint (async sharded), failure_rate = 1 per hour. T* ≈ sqrt(2 × 30 / 3600 hours⁻¹) ≈ 0.13 hours ≈ 8 minutes. In practice rounded up to 10–15 minutes to add buffer.

For W = 5 minutes (synchronous), failure_rate = 1 per hour: T* ≈ sqrt(2 × 5/60) ≈ 0.41 hours ≈ 25 minutes. The slow-write penalty extends the optimal cadence and increases lost-work per failure.

Why this matters

A team that doesn't measure failure rate ends up with arbitrary cadences ("every hour, that seems fine"). A team that measures it ends up with cadences that reflect their actual reliability surface. At 100k+ GPU scale, the cadence equation drives architecture decisions (async sharded writes mandatory, hot-standby ranks worth the cost, in-memory replication worth the engineering investment).

---

MoE and LoRA checkpoint specifics

Two architectural patterns produce checkpoint shapes that differ from dense training.

MoE checkpoints

A frontier MoE model (DeepSeek-V3 671B with 256 experts; Llama-4 Maverick with 128 experts; Snowflake Arctic with 128 experts) has most of its parameter count in experts. Expert weights are sharded along expert-parallel (EP) dimensions; each rank holds a subset of experts at rest.

Checkpoint format must include: per-expert metadata (expert ID, parent layer, EP rank assignment), the gating/router weights (dense, shared across all ranks), and the routing-state if any (for auxiliary-loss-free balancing schemes like DeepSeek-V3's bias-based approach, the bias vector is part of the state).

Reshard-on-load is critical for MoE: training might run with EP=64 across many nodes; inference might run with EP=8 on a single node; serving might use a different EP layout per region. Without reshard-on-load, you need separate offline reshard jobs.

Expert pipeline parallel adds another axis: experts pipelined across stages, each stage's experts checkpointed separately. The DeepSeek-V3 tech report describes their specific layout; the takeaway is that MoE checkpointing is inherently 2D (EP × PP) and the format must encode both.

LoRA checkpoints

LoRA (Hu et al., 2021) adds low-rank adapters on top of a frozen base model. The PEFT library's LoraConfig format is the de-facto 2026 standard: a small directory with adapter_model.safetensors (the rank-r matrices) and adapter_config.json (the config: which layers, rank, alpha, target modules).

Checkpoint size: typically 0.1–1% of the base model size. A 70B base with rank-16 LoRA on attention projections is ~50–200 MB of adapter weights. Trivial to checkpoint; the dominant cost is the base model, which is read-only and loaded once.

Multi-LoRA serving: production inference systems (vLLM, SGLang) support hot-swapping adapters per request. The adapter cache lives in GPU memory or host RAM; activation is per-request via the LoRA name in the API. The checkpoint format is identical to the training format — the same adapter_model.safetensors files.

QLoRA adds 4-bit quantization of the base model to the picture; the LoRA adapters themselves are still BF16 or FP16. Checkpoint format unchanged for the adapter; the base model's quantized form is a separate (rare) export step.

---

Checkpoint provenance and audit

In regulated industries (finance, healthcare, EU AI Act compliance), training checkpoints carry audit obligations.

Provenance metadata

Each checkpoint should be tagged with: training data version (dataset hash or version ID), code commit (full git SHA of the training script and library versions), hyperparameters (learning rate, batch size, schedule), model architecture, parent checkpoint (if resuming), training step count, wallclock timestamp, and the cluster identity. Stored alongside the checkpoint in a metadata file (JSON, YAML, or a structured manifest). The 2026 standard is one of: HuggingFace's model_card.json extensions, OpenLineage-style training-run metadata, or in-house manifests.

Reproducibility

A checkpoint with full provenance enables reproducing the run from any point. In practice "full reproducibility" is hard — RNG state, optimizer momentum, and even hardware non-determinism (NCCL all-reduce reordering, FP8 numerics) introduce drift. The realistic goal is "loss within ε of original at the same step." For audit, exact reproducibility is rarely required; demonstrating provenance and training-data linkage usually is.

Training-data linkage

Regulators increasingly require demonstrating which data trained which model version. Pattern: hash the training dataset at run start; embed the hash in checkpoint metadata; retain the data manifest separately under longer retention than the checkpoints themselves. For deduplication or PII-removal claims, the manifest includes the pre/post filtering hashes.

Retention policy

Checkpoints accumulate fast. A 1 TB checkpoint every 15 minutes for a 90-day training run is ~10 PB of raw checkpoints. Retention pattern: keep the last 10 checkpoints at full fidelity (rollback target); keep one per day at full fidelity (long-range audit); keep final checkpoint per epoch indefinitely; everything else gets garbage-collected after 7–30 days. The retained set is the audit surface.

---

In-memory redundancy patterns

The Bamboo and Check-N-Run papers explored an alternative to disk-tier checkpointing: keep recent state in the memory of other ranks. The pattern matters for preemption-heavy environments and for the very largest clusters where even staged writes are too slow.

Bamboo / pipeline-aware redundancy

Bamboo (Thorpe et al., NSDI 2023) replicates pipeline-parallel stages across spot instances such that any single preemption is recoverable from the replica without touching disk. The key insight: pipeline stages already exchange data; adding redundant copies along the existing communication paths is cheaper than building a separate replication tier. The cost is ~2× the memory footprint per replicated rank. The benefit is recovery times in seconds rather than minutes.

Check-N-Run / write-coalescing

Check-N-Run (Eisenman et al., NSDI 2022) focused on the DLRM (recommendation model) case where the embedding tables are the dominant state. The system coalesces checkpoint writes across many small updates and writes incrementally rather than full snapshots. The technique generalizes: for any state with localized updates (LoRA adapters during multi-tenant fine-tuning, sparse models with selective expert updates), incremental checkpointing reduces write volume dramatically.

GEMINI / replica-based recovery

GEMINI (Wang et al., SOSP 2023) keeps in-memory checkpoint replicas across the cluster, scheduled to minimize correlated failure risk (replicas on different power domains, different switches). Recovery from a single-rank failure is sub-second: pull state from the replica, resume. The trade-off is the memory overhead and the bookkeeping to maintain replica freshness.

When in-memory redundancy is worth it

The math: in-memory redundancy makes sense when the recovery time saved (vs. disk-tier checkpoint load) × failure rate × cluster cost exceeds the memory-overhead cost. At 100k+ GPU scale with failures every 30 minutes, recovery time savings of even 5 minutes per failure = $8k × 50 failures × extra-savings ratio = $400k-$1M over a 90-day run. Memory overhead at 2× = ~10% of training-state memory budget, which on H100/B200 hardware is modest. Net positive at frontier scale; not worth it below ~10k GPUs.

Production status

By 2026 most frontier labs run some form of in-memory redundancy alongside disk-tier checkpoints. The two complement each other: in-memory for fast single-rank recovery; disk-tier for catastrophic multi-rank loss and for long-term durability. The disk-tier checkpoint is still mandatory — in-memory state evaporates if the whole job dies.

---

Inference-side weight loading

Checkpoints don't just enable training recovery — they're the artifact that feeds inference. The inference-side loading path has its own engineering surface.

Cold-start loading

Loading a 200 GB Llama-70B checkpoint into vLLM on an 8×H100 node: ~30–90 seconds at typical NVMe + PCIe bandwidth. Loading from a network filesystem: 1–5 minutes. Loading from cold object storage: 5–20 minutes for the same checkpoint. For inference fleets that auto-scale, cold-start latency directly impacts time-to-serve when capacity must come online to handle a traffic spike.

The 2026 optimization patterns: pre-warm the local NVMe with the checkpoint on instance provisioning (Bake into the AMI / image); use Parallax-style parallel loading (concurrent reads of separate tensor shards from a parallel filesystem); use safetensors's mmap-based zero-copy load to avoid double-buffering. A well-tuned loader hits 20–40 GB/s per node, putting a 200 GB load at ~5–10 seconds.

Multi-LoRA hot-swap

The pattern in production multi-tenant serving (vLLM, SGLang, TGI): the base model loads once and stays resident; LoRA adapters swap per request. Adapter checkpoints (the PEFT format, typically 50–500 MB each) live in a cache keyed by adapter name. On request, the runtime activates the named adapter — either copying its weights into a delta tensor merged into the base, or running the rank-r matrices as a separate path.

Adapter cache management: LRU eviction with size limits; preloading common adapters on startup. The cache typically lives in host RAM (slow path to load from disk on miss) with a small subset hot in GPU memory. See multi-tenant LoRA serving for the full pattern.

Model swap

Some deployments swap entire base models. The pattern is heavier: drain in-flight requests, free the current model's GPU memory, load the new model, resume serving. Wall-clock cost: minutes to tens of minutes including the drain. Use cases: A/B testing model versions, scheduled migrations, emergency rollback of a problematic deployment.

Streaming model swaps (load the new model in parallel with the old, atomically flip routing) reduce downtime but require 2× GPU memory transiently. For frontier models that fill the GPU memory, this isn't possible without spare GPUs; the rolling-update pattern (replace one replica at a time) is cheaper.

Checkpoint quantization for inference

Most production inference doesn't use the training checkpoint directly — it uses a quantized version. The conversion (BF16 → FP8 / INT8 / INT4 with per-channel or per-block scales) is a one-time offline step that takes minutes to hours depending on model size and quantization method. The resulting checkpoint is 2–8× smaller and loads correspondingly faster.

Storing both the training-fidelity and inference-quantized checkpoints is standard. The training checkpoint is needed for resume; the inference checkpoint is what gets deployed. See the various quantization guides for the math; the storage and loading patterns are unchanged.

Tensor parallel reshard at load

Inference TP layout almost never matches training TP layout. A model trained with TP=8 PP=4 might serve with TP=4 PP=1 on smaller instances. The checkpoint format must reshard on load — DCP and Orbax both support this; legacy formats often require an offline reshard step. Plan for it: the inference team's first job after a training run finishes is usually "reshard the final checkpoint for our serving topology."

---

Checkpoint compression, deltas, and quantized states

The 14 TB number from the worked example is a lot of bytes. Several patterns reduce it.

Quantized optimizer state

Adam's m and v buffers are 32-bit floats by default — 8 bytes per parameter combined. 8-bit Adam (Dettmers et al., 2022) quantizes both to 8 bits with block-wise scales, cutting optimizer-state footprint by 4× with no measurable training quality regression. Standard in the bitsandbytes library; common in low-memory training and large-model training where the optimizer footprint is the bottleneck. Checkpoint savings track the memory savings: 12 TB → 3 TB on the worked-example model.

Delta checkpoints

Instead of saving the full state every cadence, save the difference from the last full checkpoint. Pattern: full checkpoint every Nth cadence; deltas in between. Storage scales with the per-step parameter change rate, which for late-stage training is small.

Reconstruction: load the latest full checkpoint, then apply each delta in sequence. Recovery time grows with the number of deltas; the trade-off is more storage savings at the cost of slower recovery.

Production use is mixed — the engineering complexity (delta encoding, validation, atomic application) often isn't worth it below frontier scale.

Sparse encoding for MoE

In an MoE checkpoint, only the active experts in any given step change much; the inactive ones drift slowly. Sparse delta encoding (save only experts that changed by more than a threshold) cuts MoE checkpoint write volume significantly. DeepSeek-V3's tech report describes their per-expert update patterns; the storage savings inform their checkpoint cadence.

Gradient compression

Not strictly a checkpoint optimization, but related: in distributed training, gradient compression (1-bit Adam, PowerSGD, top-k sparsification) reduces all-reduce bandwidth. The trade-off is reduced precision in gradient updates; mostly used in bandwidth-constrained training (cross-datacenter, slow interconnects). Less common in frontier training where InfiniBand bandwidth is abundant.

Compression at write

Plain zstd compression on checkpoint shards at write time cuts storage 30–60% for typical weight tensors. CPU cost: a single core sustains a few hundred MB/s of zstd-1 compression; for terabyte checkpoints with many cores available, the compression overhead is negligible. Worth doing for any checkpoint headed to durable storage; less useful for the hot staging tier where the speed gain doesn't justify the CPU.

Comparison

Technique	Storage savings	Recovery time impact	Engineering complexity
zstd compression	30–60%	+5–15% load time	Low (one library call)
8-bit optimizer	4× on optimizer state	None (transparent)	Low (bitsandbytes integration)
Delta checkpoints	5–20× between fulls	Linear in delta count	High
Sparse MoE deltas	Depends on activity	Moderate	High
FP16 master weights instead of FP32	2× on master weights	None at training; risks at fine-tuning	Low but tradeoff-sensitive

---

Recovery runbook patterns

A failure happens. What does the on-call engineer actually do, in order?

Detection (0–2 minutes)

Liveness probes on the training ranks fire. The orchestrator (TorchElastic, Ray Train, MosaicML Composer, the in-house equivalent) detects the membership change. Alerts fire to on-call. The first signal is usually "the loss curve stopped advancing"; the second is a Slack ping from the orchestrator's failure detector.

Triage (2–10 minutes)

On-call pulls up the dashboard. Which rank(s) failed? What's the failure signature (OOM, network drop, NCCL timeout, GPU XID error, hardware fault flag)? Decision tree:

• Single rank, hardware-suspect: quarantine the rank, schedule replacement from warm spare pool, expect 5–10 minute resume.
• Single rank, transient (network blip, software hang): kill the rank, let the orchestrator restart it, expect 2–5 minute resume.
• Multi-rank, correlated (rack PDU, switch, storage): investigate infrastructure cause before resuming. Don't just restart — you'll lose the next checkpoint too.
• Soft failure (loss spike, NaN, gradient explosion): stop the job, examine recent checkpoints for SDC, possibly rollback to an earlier good checkpoint.

Recovery (5–15 minutes for healthy case)

The orchestrator (with help of the on-call if escalation needed): load the latest validated checkpoint, redistribute across the new topology, resume training. The TorchElastic / Ray Train / Composer abstractions handle most of this automatically once the failing rank is replaced.

Verification (5 minutes post-resume)

After resume, watch the loss curve for 5–10 minutes. If it's smoothly continuing the prior trajectory, the recovery worked. If it diverges, suspect SDC or a misloaded checkpoint and consider rolling back further.

Postmortem (within 1 week)

Every recovery event gets a brief postmortem: what failed, how was it detected, how long did recovery take, what could be improved. The patterns that emerge over many incidents drive infrastructure investment — if 30% of failures are storage-related, invest in storage diversity; if recovery takes 30+ minutes routinely, invest in faster reshard-on-load.

Common anti-patterns

• Restarting blindly: the failure mode that destroyed one rank may destroy the replacement too. Investigate before retrying.
• Skipping checkpoint validation: resuming from a corrupted checkpoint is worse than starting earlier from a good one.
• No on-call playbook: every recovery starts from scratch. Document the decision tree, the rollback procedure, the escalation contacts.

---

Chaos engineering for training

Recovery drills (covered in the drills section) are scheduled exercises. Chaos engineering is the practice of continuously injecting failures to validate the recovery path stays working.

What to inject

• Rank kill: SIGKILL a training rank at random. Validates per-rank recovery.
• Network partition: simulate a NIC drop or switch loss. Validates the collective failure detection and rendezvous path.
• Storage degradation: throttle the checkpoint write bandwidth to 10% of normal. Validates async write headroom and timeout handling.
• Slow rank: inject latency on one rank's compute. Validates straggler detection and (if implemented) skip-the-straggler logic.
• Silent corruption: flip bits in a written checkpoint shard. Validates checksum-based corruption detection.

Frequency

Continuous in production-critical clusters. Weekly or per-run in less-critical environments. The discipline: every failure mode you've ever seen in production should have a corresponding chaos injection that exercises the recovery path.

Tooling

Most large AI labs build internal chaos-injection tooling on top of Kubernetes' chaos-mesh, or run scripted kills via a control-plane API. Litmus, Chaos Monkey, Gremlin, and other classic chaos tools work too but are designed for service-style workloads; training-specific tooling typically wraps them.

When chaos finds bugs

It will. The bugs are typically: race conditions in the recovery path that only surface under load; missing timeouts that hang indefinitely on partial failure; replication paths that silently fall behind; metric pipelines that don't actually fire on the conditions you thought they did. Each finding is a fix you needed anyway; better in chaos than in real failure.

---

SDC mitigation deep dive

Silent data corruption (SDC) at scale is documented in Meta's SDC papers (arXiv:2102.11245, arXiv:2204.00455) and Google's CPU corruption studies. The rates are non-trivial: ~1 in 1000 CPUs exhibits some SDC over its lifetime; GPU rates are less-studied but likely similar order of magnitude.

Detection strategies

• All-reduce verification: in distributed training, the same gradient is computed independently on multiple ranks (data parallelism replicates the work). Comparing all-reduced results across DP groups catches a subset of SDC — if one rank's gradient differs and a sanity check (e.g., gradient norm) flags the outlier, that rank is suspect.
• Periodic deterministic replays: every N steps, re-run the same step on a different rank set with the same RNG seed. Compare loss; significant divergence signals SDC somewhere in the original run.
• Per-shard checksums: BLAKE3 or SHA-256 on every checkpoint shard at write time; verify on read. Catches storage-tier corruption.
• ECC monitoring: GPU ECC correctable-error rates spike before uncorrectable errors. Aggressive monitoring + quarantine of suspect GPUs prevents many SDC incidents.
• Loss-curve anomaly detection: a sudden spike in loss or gradient norm that doesn't correlate with hyperparameter changes is a smell. Investigate before assuming a data issue.

Mitigation strategies

• Quarantine and replace: a GPU flagged via ECC, comparison, or replay is removed from the active pool, and a hot spare takes its place. Replacement cost: minutes; alternative cost: hours-to-days of suspect training.
• Hot spares: maintain a pool of warm GPU nodes ready to drop into the cluster. The pool size is sized to expected failure rate × replacement time.
• Diverse hardware vintages: a single bad SKU or batch can cause clustered failures. Mixing GPUs from different production runs / vendors reduces correlated-failure risk.
• Redundant compute on critical paths: for the most sensitive operations (e.g., reward model computations in RLHF, where one bad reward can derail training), redundant computation with cross-rank comparison.

Real failure rates

Anecdotal numbers from public postmortems and the DeepSeek-V3 tech report: at 100k+ GPU scale, expect a handful of SDC-suspected failures over a multi-month run. Each one costs hours-to-days of investigation, sometimes triggering a rollback to a pre-corruption checkpoint. The cumulative cost of SDC, if not actively mitigated, can be weeks of training time over a frontier-scale run.

---

Worked example: a 100k-GPU run's checkpoint budget

Bringing together cadence, throughput, and storage cost for a frontier-scale training run.

Setup

• 100,000 H100 GPUs across 12,500 nodes (8 GPUs/node).
• Model: 1T-parameter dense (for simplicity; MoE math is similar but the active param count differs).
• BF16 weights: 2 bytes × 1e12 = 2 TB.
• Optimizer state (Adam: m, v, FP32 master weights): 3 × 4 bytes × 1e12 = 12 TB.
• Total per-checkpoint state: ~14 TB.
• Sharded across 100k GPUs: ~140 MB per rank.

Cadence

Failure rate (effective) ≈ 1 per 30 minutes at this scale. Cadence equation with W = 30s async sharded write: T* ≈ 5 minutes. Round to 10 minutes for buffer. Expected lost work per failure: ~5 minutes of training time, which at $100k/hour cluster cost is ~$8,000 per failure. Cheaper than the alternative.

Write throughput

Per-rank write: 140 MB to local NVMe + replicate to cluster FS. Per-node aggregate: 8 × 140 MB = ~1.1 GB. Cluster aggregate per checkpoint: 14 TB written, replicated, ~28 TB if including replication overhead. At 30s wall-clock: 14 TB / 30s ≈ 467 GB/s sustained to durable tier. Achievable on a properly provisioned Lustre or WekaFS deployment.

Storage cost

14 TB per checkpoint × 6 checkpoints/hour × 24 hours × 90 days = ~180 PB of raw checkpoint data over a 90-day run. With retention policy (last 10 + one per day + final per epoch), live storage is ~2–5 PB. At cloud-tier prices ($20/TB/month for performance object storage), that's $40–100k/month for storage alone. At training-tier (parallel FS) it's higher per TB but lower volume (live working set is smaller).

Recovery

Failure detected within 1–2 minutes by liveness checks. Replacement ranks scheduled within 5 minutes (warm spare pool). Checkpoint load from cluster FS: 14 TB read in parallel by 100k ranks, ~30 seconds wall-clock. Resume training within ~7 minutes of failure. Total recovery tax per failure: ~12 minutes of cluster time, or ~$20k at $100k/hour. Multiply by ~50 failures over a 90-day run: $1M of recovery tax. Reasonable insurance against the ~$200M training cost of the run.

Sensitivity

Cut cadence to 30 minutes (less aggressive): expected lost work per failure becomes 15 min; total wasted work over 50 failures is 12.5 hours = $1.25M. Doesn't sound terrible until you remember that the write itself was supposed to be cheap (30s × 6/hour = 3 min/hour overhead = 4.3 hours over 90 days = $430k). So the "fast cadence" plan costs ~$430k in stall but saves ~$1M in lost work; net win.

Increase checkpoint write time to 2 minutes (e.g., synchronous, badly provisioned storage): the cadence-equation optimum jumps to ~24 minutes; lost work per failure climbs accordingly. The fast-write infrastructure pays for itself in compute saved.

---

Checkpoint security: encryption, RBAC, and audit

Checkpoints are the most valuable artifact your training pipeline produces — they are the model. Treating them as ordinary build artifacts in an unrestricted bucket is the same posture as committing production credentials to a public repository. By 2026, the bar for production training infrastructure has moved decisively toward encryption-at-rest with cluster-managed keys, role-based access to checkpoint paths, and append-only audit logs of every read and write.

Encryption at rest

The basic question is where you do the encryption. Three layers, each with different trade-offs.

• Storage-layer encryption (server-side encryption on S3, GCS CMEK, Azure SSE) is the cheapest to deploy — the storage backend transparently encrypts blocks with a customer-managed key (CMK) brokered by a KMS. Recovery requires the recovery job to have the IAM/role grants to invoke the KMS. Zero application-side code change. Weakness: the storage operator can be compelled to decrypt; the data is not encrypted from the application's perspective, only from the disk's.
• Filesystem-layer encryption (LUKS on NVMe, parallel-FS-level encryption on Lustre/WekaFS) protects against physical-disk theft and operator-level snooping at the storage tier. Compose with storage-layer encryption for defense in depth.
• Application-layer encryption is the strongest posture: the training job encrypts each shard with a key derived from a cluster KMS before writing. The storage never sees plaintext. Recovery requires the same KMS access. The performance cost is small with hardware-accelerated AES (AES-NI on x86, ARMv8 crypto extensions, NVIDIA NVEnc/CryptoAPI on Hopper/Blackwell) — typically 1–3 GB/s/core, far below NVMe write speed. Use a separate Data Encryption Key (DEK) per checkpoint shard, wrapped by a Key Encryption Key (KEK) from KMS; rotate the KEK without rewriting shards.

For confidential-computing-aware deployments (H100 CC, Blackwell TEE, the upcoming intra-rack attested NVLink fabrics), the standard pattern is to have keys released to the TEE only after remote attestation succeeds. The training job cannot exfiltrate plaintext checkpoints unless the attestation surface is broken.

Role-based access control

A production checkpoint bucket has at minimum four roles:

• Trainer-writer — the training job's service account. Write-only to the active run's directory; no read on prior runs.
• Validator-reader — the validator/eval job's service account. Read-only on a sampled subset of paths; no write.
• Recovery-reader-writer — the recovery operator's role. Read on all paths, write to the recovery target. Audited.
• Publisher — the path that pushes selected checkpoints to a public or partner-facing path. Read-only on the curated set.

Cross-role escalation should require a break-glass procedure with on-call sign-off. Treat the path that contains a billion-dollar training run's weights as the production crown jewel; the people who can read it should fit in a small Slack channel.

Audit logging

Every read and write should land in an append-only audit log (CloudTrail, GCP Audit, Azure Monitor, plus an internal SIEM). The log should answer two questions in seconds: who read this checkpoint, when, from where? and what writes happened to this run's directory in the last 24 hours? In practice, the second question becomes the early-warning signal for "an automated job is overwriting checkpoints faster than expected" or "the validator is failing silently and falling behind."

Threat model checklist

Threat	Mitigation
Insider exfiltration	Application-layer encryption + per-role decryption + audit logging
Bucket misconfiguration	Default-deny IAM, periodic config scans, blocked public access at org policy
Compromised training-job token	Short-lived credentials (1h max) issued via OIDC, no long-lived service-account keys
Supply-chain attack on checkpoint loader	Pinned safetensors-only loaders, no pickle on production paths, signed checkpoints
Silent overwrite of latest pointer	Object-versioning + MFA-delete on the bucket, atomic rename through cluster FS
Cross-region replication leak	Replication target with same encryption + same IAM; replication-status alerts

The threats are not hypothetical: by 2025 there were public incidents of unreleased model weights surfacing on torrent sites, traced back to over-permissive checkpoint paths. Lock them down.

---

Checkpoint versioning: DVC, MLflow, and metadata stores

A checkpoint without metadata is a file with weights in it. A checkpoint with metadata is an artifact with provenance — which data subset trained it, which code revision produced it, which hyperparameters were in flight when it was written, which evals it passed. By 2026 the metadata-store ecosystem has consolidated around a few patterns.

DVC (Data Version Control)

Open-source, git-adjacent. DVC stores small pointer files in git that reference large artifacts in object storage; the pointer encodes the content hash and the remote path. Strengths: dev-loop ergonomics, branch/merge semantics on data, language-agnostic. Weakness: not built for the operational concerns of frontier training (no built-in eval lineage, no real-time write tracking). Fits research and small-team production.

MLflow

The Apache-licensed reference. The tracking server stores experiment runs (hyperparameters, metrics, code git-SHA) and the artifact store keeps the checkpoint blobs. Strong for tabular metric tracking; weaker on the actual artifact lifecycle (it stores them, but doesn't enforce retention or replication policy). Heavily used inside Databricks and similar environments.

Weights & Biases (W&B) Artifacts

W&B's artifact store treats each checkpoint as a versioned object with an explicit lineage graph (which run produced it, which run consumed it). Strong UI. Production-grade at the scale of most enterprise training, though frontier labs typically build their own equivalent on top of object storage + a metadata catalog.

Internal metadata catalogs

Frontier labs typically run an internal Postgres-or-Spanner-backed catalog over their object-store checkpoint paths. Schema includes:

• Checkpoint URI (path in cluster FS + object store)
• Content hash (SHA-256 of all shard hashes)
• Step number, epoch, wall-clock timestamp
• Training-run ID (foreign key to the run record)
• Code git-SHA, container digest, framework versions
• Hyperparameters snapshot
• Eval results (link to eval-run rows)
• Compliance flags (training-data lineage, regulatory tags)
• Retention class (transient / weekly / quarterly / permanent)

The catalog is the source of truth; the storage is just bytes. When the catalog and storage disagree, the catalog wins — the storage gets reconciled.

Comparison

System	Best for	Frontier-scale fit	Open-source
DVC	Research, small teams	Limited	Yes
MLflow	Experiment tracking + artifacts	Mid-scale	Yes
W&B Artifacts	Enterprise training	Most enterprise	No
Internal catalog	Frontier labs	Built for it	Custom
Hugging Face Hub	Published / shared weights	Publishing only	Partial

---

MosaicML Composer and the training-orchestrator angle

MosaicML's Composer (now part of Databricks) is one of the better-engineered open-source training orchestrators specifically focused on reliability and checkpoint hygiene. It's worth a section because its design choices represent the consensus 2026 pattern even for teams that don't use it.

Composer's checkpoint design hits several patterns at once: async sharded writes via DCP, atomic finalization via tmp+rename, configurable retention (keep last N + every Mth + final), per-shard checksums, automatic resume on restart, and elastic-friendly reshard-on-load. The composer.callbacks.CheckpointSaver API exposes cadence, retention, and target-store as first-class config rather than implementation detail.

Equally important: Composer ships with a runtime that wraps the training loop in a fault-tolerance layer (timeouts, NCCL watchdog, automatic local-rank restart on transient failure). The combination of checkpoint hygiene + supervisor-layer recovery is what makes Composer production-friendly out of the box, where a hand-rolled PyTorch loop typically takes engineer-months to reach the same posture.

Comparable frameworks: Lightning AI's PyTorch Lightning has analogous patterns (Trainer(strategy="fsdp") integrates with DCP); Ray Train provides an elastic-runtime orchestrator; Determined AI offers a managed alternative. The choice between them is largely an integration question rather than a capability one — all of them can drive a multi-thousand-GPU training run with sharded async checkpointing and supervised recovery.

---

The Llama 3.1 405B reliability postmortem in detail

Meta's Llama 3.1 paper (2024) included one of the most candid reliability accounts in the published frontier-training literature. The training cluster was approximately 16,000 H100 GPUs running for ~54 days; the paper documents the failures across that window. The breakdown (rough, from the paper's tables — see the Llama 3 technical report for exact numbers): a sizeable majority of interruptions were attributable to GPU hardware (HBM ECC double-bit errors, GPU thermal events, NVLink failures), followed by host-side issues (DRAM errors, network NIC events), and a long tail of software (NCCL hangs, framework-level bugs).

The take-aways that generalize:

• Failure rate is dominated by GPU-side hardware. Provisioning a hot-spare pool sized to 5–10% of the cluster is the practical defense; failures are frequent enough that without it, recovery latency dominates run time.
• NVLink and NIC failures are the worst class. Unlike a GPU thermal event (which is local), a fabric failure can affect dozens of ranks simultaneously and triggers an entire rack's pause. The Llama 3 team invested heavily in rack-level isolation so that a single fabric event was a "lose one rack" event rather than a "lose the run" event.
• The mean-time-to-recovery (MTTR) was the headline metric, not the mean-time-between-failures (MTBF). Checkpoint cadence was tuned so that expected lost work per failure was on the order of single-digit minutes; recovery itself was automated end-to-end via the runtime.
• A significant fraction of interruptions did not require intervention. Automatic restart from latest checkpoint handled the majority; manual intervention was reserved for failure modes that the supervisor didn't recognize (silent corruption, recurring straggler patterns).

The paper's most repeated lesson: build the reliability stack before you train at scale, not during. Retrofitting a fault-tolerance layer onto a hand-rolled training loop in the middle of a 50-day run is how runs slip a quarter.

---

Cloud-native multipart strategies (S3, GCS, Azure)

When the checkpoint target is object storage rather than a parallel filesystem, the multipart-upload semantics of the underlying cloud determine throughput. The numbers below are 2026 best-case for well-provisioned accounts; per-region and per-account variation is large.

AWS S3

Per-connection PUT bandwidth peaks around 100 MB/s; multipart upload with 16 MiB parts and dozens of concurrent parts can sustain 5–20 GB/s per node. S3 Express One Zone (single-AZ, low-latency) shaves latency by ~10× for small parts but trades durability — only appropriate for staging tiers, never for archival. The published soft limit of 3,500 PUT requests/second per prefix is the bottleneck for many-small-shard checkpoints; the workaround is to scatter shards across prefixes (/run42/r0001/..., /run42/r0002/...).

GCS

Per-stream upload tops out at ~150 MB/s; XML multipart upload, parallel composite uploads, and the gRPC-based Storage API push aggregate per-node throughput to similar 5–20 GB/s. The "composite object" pattern (upload many smaller objects, compose them server-side) is GCS-specific and useful for very large shards. Account-level egress quotas matter at frontier scale; coordinate with your TAM.

Azure Blob

Block blobs with high-throughput tier; per-stream ~150 MB/s, aggregate higher with parallelism. Azure's hot/cool/archive tier semantics map naturally onto the retention class hierarchy (live → hot, weekly → cool, quarterly → archive).

Multipart-upload patterns to know

• 16 MiB parts — the consensus sweet spot. Smaller parts hit per-request rate limits; larger parts have higher retry-cost on transient failures.
• Hundreds of parallel parts — the only way to hit the aggregate bandwidth.
• Integrity headers — Content-MD5 per part, full-object SHA-256 in metadata. Object stores will reject corrupted parts on PUT if you include the hash.
• Lifecycle policies — automated transition from hot to cool to archive based on age. Avoid manual retention management.
• Cross-region replication — separate IAM, separate encryption keys; replicated objects are not automatically encrypted with the source key.

Comparison

Cloud	Per-stream (MB/s)	Per-node aggregate (GB/s)	Strong point	Weak point
S3	~100	5–20	Mature, broad IAM	Per-prefix rate limit
GCS	~150	5–20	gRPC API, composite	Account quotas
Azure Blob	~150	5–20	Tier semantics	Region-specific quirks

---

Async checkpointing libraries: TorchSnapshot, NVIDIA Resiliency

Beyond the framework-native DCP/Orbax/DeepSpeed paths, a few libraries are worth mentioning by 2026:

• TorchSnapshot — Meta's earlier async-checkpoint library, mostly subsumed by DCP for new code but still in production at Meta. Pre-dates DCP and influenced its design; the async-and-sharded patterns originate here.
• NVIDIA Resiliency Extension — NVIDIA's training-resiliency library that integrates with NeMo, providing automatic failure detection, rank replacement, and checkpoint reload. Roughly the NVIDIA-shop equivalent of TorchElastic + DCP.
• DeepSpeed Async Checkpoint — DeepSpeed-native async support that drains state to NVMe in the background. Comparable to FSDP2 + DCP for ZeRO-based training.
• Apex DistributedCheckpoint — older NVIDIA library, mostly historical now.
• JAX Orbax checkpointers — the canonical JAX-side library; supports async by default and tight integration with jax.distributed.

The headline pattern across all of them is the same: foreground host-RAM copy (sub-second to seconds), background flush to durable tier, atomic finalization, retention-managed. The differences are framework integration and operational ergonomics, not algorithmic.

---

Erasure coding vs replication: the cost-curve math

For in-memory or NVMe-tier redundancy across ranks, the choice between full replication (Bamboo/GEMINI-style) and erasure coding (Reed-Solomon-style) is a fixed-overhead-versus-CPU-cost trade-off.

• Full replication (2× or 3×) stores 2–3 full copies of each shard on different nodes. Overhead: 100–200% storage. Recovery on single-node failure: trivial — read from any replica. CPU cost: zero. Failure tolerance: tolerates replicas − 1 simultaneous failures.
• Reed-Solomon (k, m) stores k data blocks plus m parity blocks; tolerates any m simultaneous failures. Overhead: m/k storage. Recovery: read k blocks (data or parity), reconstruct missing. CPU cost: encode on write, decode on recovery — typically 1–5 GB/s per core with AVX-512 or Galois-field instructions. Failure tolerance: tunable via m.

For checkpoint use cases where the shards are large and failures are uncorrelated, RS(8,2) — 8 data, 2 parity — gives 25% storage overhead with 2-failure tolerance. Compare to 3× replication's 200% overhead. The trade-off is CPU work on encode (which can run async after the staging copy) and on decode (paid only at recovery, when wall-clock matters most). For frontier deployments where storage cost dominates and CPU is cheap, RS coding wins; for environments where recovery latency dominates and CPU is precious, replication wins.

Pattern	Storage overhead	CPU overhead	Failure tolerance	Recovery latency
2× replication	100%	None	1	Read-1
3× replication	200%	None	2	Read-1
RS(8, 2)	25%	Encode + decode	2	Read-8, decode
RS(10, 4)	40%	Encode + decode	4	Read-10, decode
RS(20, 4)	20%	Encode + decode	4	Read-20, decode

---

Resharding deep dive: how DCP's planner actually works

PyTorch DCP's reshard-on-load capability is one of the more underappreciated 2026 features; it's the reason FSDP2-trained checkpoints can be moved between TP/PP/DP layouts without manual surgery.

The mechanism: a checkpoint written via DCP contains a .metadata file that describes, for each logical parameter, the global shape and the per-shard byte offsets. The reader is initialized with the current topology (TP=8, PP=4, DP=N). DCP's planner computes a load plan that, for each rank in the current topology, maps which shards in the saved checkpoint contain the bytes that rank needs. The plan can split across files, gather from multiple shards, or read a strict subset — whichever serves the destination topology.

Key implications:

• Save in the topology you have, load in the topology you want. Useful when promoting a checkpoint from a 1024-GPU run to a 4096-GPU continuation, or when downscaling for fine-tuning.
• Reshard works for FSDP2 and TP/PP combinations natively; for ZeRO-3 you typically go through DeepSpeed's zero_to_fp32.py consolidation step first.
• Metadata must be saved correctly. A checkpoint with a missing or stale .metadata is effectively unloadable in a different topology. Treat the metadata file as more important than any individual shard.

The planner is open-source PyTorch; reading its source is a good way to understand the format. The lesson generalizes: a checkpoint format that records what each shard is alongside the bytes — not just which rank wrote it — is what enables flexible recovery.

---

The bottom line

The problem is the recovery tax: at frontier scale, failures are not exceptional events — they are a baseline against which every minute of unbacked training time is debited. The solution is async sharded checkpointing with atomic finalization, tiered storage, and a recovery path that has been exercised before it's needed. The biggest lever is moving from synchronous to async sharded writes; that single change typically lets you checkpoint 4–8× more often at the same overhead.

• Pick cadence by failure rate, not habit. Expected lost work per failure should roughly equal checkpoint overhead.
• Atomic finalization is non-negotiable. Write to tmp, fsync, rename — anything else risks a corrupt "latest" pointer.
• Tier your storage. RAM staging → local NVMe → cluster FS → object store. Different durability, different cost.
• Checksum every shard. Silent corruption is real at scale; without checksums you don't know which checkpoint is valid.
• Run recovery drills. A checkpoint you've never restored from is not a checkpoint; it's an unvalidated file.

For the parallelism layout that determines how your state shards, read distributed LLM training; for the network bandwidth your write path actually has, read NVLink and rack-scale topology.

---

FAQ

How often should I checkpoint? For a typical production run with hourly-scale failure rates: every 15-30 minutes. Calibrate against your failure rate and checkpoint duration.

Should I checkpoint to S3 directly? Usually no. S3 per-stream bandwidth is low; you'd block training for too long. Write to a fast tier first, async-replicate to S3.

Can I skip optimizer state to save space? You can, but resuming requires re-initializing the optimizer, which loses recent momentum/variance. Bad idea for ongoing training; OK for archival or for handoff to fine-tuning that may not need the optimizer state.

What if my framework doesn't support sharded checkpoints? Either limit your model size or invest in tooling. For frontier-scale training, sharded checkpoints are essentially mandatory.

How do I test recovery in production? Schedule periodic recovery drills: kill a process, verify recovery, check metrics. Better to discover bugs in drill than in real failure.

Do I need to checkpoint during inference deployments? For weights and KV-cache state, no (weights are read-only and loaded at startup; KV is per-request). For warm-pool sandbox state in agent systems, yes — with much smaller and simpler checkpoint requirements.

What's the right checksum algorithm? SHA-256 or BLAKE3. Don't use MD5 (collision risk) or skip checksumming entirely.

How big should my staging buffer be? Big enough for the largest single checkpoint. Sized in host RAM. For terabyte-scale checkpoints, this is real memory commitment.

How does silent data corruption actually present itself? Usually as a slow, mysterious divergence in training loss that can't be traced to a hyperparameter or data change. Sometimes as a sudden NaN that doesn't recur on restart. The Meta and Google CPU-SDC papers documented the phenomenon on CPUs; GPUs have similar (less-studied) failure modes. Defense in depth: per-rank loss/gradient-norm monitoring, periodic redundant computation on a sampled subset of ranks, and the discipline to quarantine and replace a suspect GPU rather than hope.

How does fault tolerance interact with elasticity (TorchElastic, Ray Train)? Elastic frameworks let you add or remove ranks mid-run. They depend on a working checkpoint-and-restart path; you don't get elasticity without reliability. Used together they enable partial recovery — replace only the failed ranks instead of restarting everything — which is the practical 2026 default at large scale.

Do I need separate checkpoints for the model and the optimizer? Conceptually no — they're both part of training state. Operationally yes, sometimes: keeping the model weights in a portable format (safetensors) alongside the framework-specific optimizer state lets you publish or hand off the model without dragging the optimizer state along. Many production setups save both per-step.

What's the right monitoring stack for checkpoint health? At minimum: last-good-checkpoint age, write latency p50/p99, write success rate, storage utilization per tier, replication lag to durable tier. Alert on age (no checkpoint in 2× cadence), write failures, and saturation. Dashboards beat manual checks; pages beat dashboards.

How does checkpoint design interact with synthetic data and distillation pipelines? Distillation runs produce teacher-generated data alongside student training; the teacher's outputs may need to be checkpointed too (especially for resume-with-same-data semantics). For most setups the teacher is frozen and the only state worth checkpointing is the student — but if you're doing online generation, treat the generator's RNG and step counter as first-class checkpoint state.

Is in-network checkpointing (writing during collectives) worth the engineering investment? Only at frontier scale. Below ~10k GPUs, async checkpointing to host RAM + background flush gets you to single-digit-second stall, which is plenty. In-network checkpointing pays off when the marginal compute hour saved is more valuable than the engineer-quarter to build it — i.e. very large runs where every 1% of throughput is six figures.

What's the difference between safetensors and the older pickle-based format? safetensors is a memory-mapped, type-safe, language-agnostic tensor file format that loads faster and doesn't suffer from pickle's arbitrary-code-execution risk. For weight checkpoints meant to be published or moved across frameworks, safetensors is the 2026 default. For framework-native checkpoints (sharded state with framework-specific optimizer state), the framework's binary format is fine — but safetensors-format weight copies alongside are increasingly standard practice.

Should I use object storage (S3) for live checkpoints or only for archival? Archival only, in almost all cases. S3 per-stream bandwidth is too low to be the primary checkpoint target during training — you'd block training waiting for the write. Pattern: write to parallel FS (Lustre / WekaFS / GPFS) for live recovery, async-replicate to S3 / GCS for durability and multi-region failover. S3-as-primary works only at very small scale or for sparse checkpoint cadences where the bandwidth gap doesn't bite.

How do MoE checkpoints differ from dense-model checkpoints? MoE adds expert weights and routing-state. Expert weights are usually the dominant size term — a 671B MoE model with 256 experts has most of its parameter count in experts. Expert-parallel layouts mean each rank holds only a subset of experts; sharded checkpoints reflect this. The checkpoint format must include per-expert metadata (expert ID, expert-parallel rank assignment) so a different EP layout can reshard on load. See MoE serving for the EP layout that shapes the checkpoint structure.

What's the right way to checkpoint during post-training RLHF/DPO? RLHF/DPO runs add a reward model (or preference data) to the state. Checkpointing best practice: separate the policy checkpoint (the model being trained, same format as pretraining) from the reward model (often a separate frozen checkpoint loaded at start of run) and the replay buffer / preference dataset (versioned alongside but typically not in the same file). Resume from a policy checkpoint + reload reward model + resume from saved replay-buffer cursor.

How do checkpoint formats handle FP8 training state? With care. FP8 training (FP8 training tradeoffs) typically keeps a master copy of weights in BF16 or FP32 alongside the FP8 weights. The checkpoint must include the master weights — losing them on resume causes drift from re-quantization noise. DeepSeek-V3's tech report (arXiv:2412.19437) documents their specific layout. Frameworks that auto-handle this (transformer-engine, Megatron-LM with FP8 support) save both; rolling your own FP8 training requires care to checkpoint both.

What does checkpoint sharding look like for FSDP vs ZeRO-3? Conceptually the same — each rank holds and saves a shard of the model and optimizer state. Implementation differs: FSDP (PyTorch native) ships well-integrated with DCP; ZeRO-3 (DeepSpeed) has its own checkpoint manager that handles the same sharding pattern. For PyTorch-native stacks, FSDP + DCP is the cleanest path; DeepSpeed users have their own tooling that's also production-ready.

How do I migrate a checkpoint from one parallelism layout to another? Two options. (1) Consolidate to a topology-independent format (gather all shards to rank 0, write one big file, redistribute on load). Slow but reliable. (2) Use DCP's reshard-on-load planner, which can convert sharded checkpoints between TP/DP/PP degrees on the fly. Faster but requires the planner to understand your model's layout.

What's the role of synthetic data and distillation in checkpoint design? Distillation runs often produce teacher outputs alongside student training. The state you might want to checkpoint includes: the student's training state (standard), the teacher's frozen weights (load once, no need to checkpoint mid-run), and the synthetic data cursor (if you're generating data online). Most setups treat the teacher as immutable infrastructure and only checkpoint the student.

Can I run checkpoint validation as a continuous background job? Yes, and frontier labs do. The pattern: a separate "validator" job runs on spare capacity, periodically loads recent checkpoints, validates their integrity (checksums, shape, basic forward pass to non-NaN loss), and reports to the monitoring system. Detects checkpoint corruption proactively rather than at recovery time. Cheap insurance.

What happens if the cluster filesystem itself fails mid-write? Atomic finalization saves you — the in-progress checkpoint is in a .tmp directory and gets ignored on recovery. Falling back to the prior valid checkpoint loses ~one cadence-worth of work. The catastrophic case is filesystem corruption that takes down multiple recent checkpoints; defense is multi-tier (NVMe + parallel FS + object store) with diverse failure modes.

Is there ever a reason to not checkpoint optimizer state? Yes, for handoff or archival where the receiver will fine-tune from scratch or with a different optimizer. Saves 4× the storage. For continuing the same training run, never — restarting Adam's momentum and variance from zero loses substantial training progress and effectively wastes the recent training.

What's the practical bandwidth cap on writing checkpoints with multipart S3 uploads? Per-connection: ~100 MB/s on AWS S3, ~150 MB/s on GCS, varying on Azure Blob. Aggregate with parallel multipart uploads (dozens of concurrent connections, 16 MB part size): 5–20 GB/s achievable per node, 50+ GB/s with many nodes. The bottleneck is usually the egress NIC and the per-account request rate limits, not the object store itself. For terabyte-scale checkpoints, plan multipart uploads with hundreds of parallel parts and use S3 Transfer Acceleration or equivalent.

How do I handle checkpoint compression for sparse training (MoE, sparse-attention)? Sparse checkpoints save substantial space when much of the state is zero. Standard pattern: COO/CSR-style sparse encoding plus per-shard compression (zstd at level 1–3 is the cost/benefit sweet spot). MoE checkpoints don't benefit much because expert weights are dense; sparse-attention training (Mamba-style state space models, sparse-mixture variants) does. Delta checkpoints (storing only the changes since the last full checkpoint) are another option, but the bookkeeping is non-trivial and the savings depend on workload.

Can I checkpoint to multiple regions for disaster recovery? Yes; the pattern is async replication from the primary tier (cluster FS) to a secondary region (object store with cross-region replication enabled). Recovery from a regional outage: pull the latest replicated checkpoint, redeploy the training job in a different region. The catch is that replication lag (typically 5–30 minutes for object-store cross-region) bounds how recent the secondary copy is. Acceptable for catastrophic-region-loss scenarios; not a substitute for in-region rapid recovery.

How do I detect silent data corruption proactively, not just at recovery time? Per-shard checksums computed at write time and verified periodically. A background validator job samples checkpoints, recomputes checksums, and alerts on mismatches. For deeper validation, periodically load a sampled checkpoint into a small validation cluster and run a forward pass to confirm loss is non-NaN and within bounds. Costly but catches the worst SDC modes — corrupt-yet-valid-looking checkpoints — before they cause a training disaster on recovery.

What's the lost-work cost of a 1-hour cadence vs a 15-minute cadence at 100k-GPU scale? With effective failure rate of 1/hour (a conservative number at 100k GPUs): 1-hour cadence loses 30 minutes per failure on average; 15-min cadence loses 7.5 minutes. At cluster cost of ~$100k/hour, the difference is ~$37k per failure. Over a 90-day run with ~2000 failures, that's $75M difference. The write-overhead difference (4× more checkpoints) is far smaller: ~30s × 4 extra writes per hour × 2160 hours = 2160 minutes = $3.6M. Net savings of fast cadence: ~$70M. Numbers shift with your actual failure rate and cluster cost, but the direction holds.

How does TorchElastic interact with the checkpoint format? TorchElastic supports elastic agents that join and leave the training group. On membership change, the framework triggers a re-sharded checkpoint load: existing ranks save their state, new ranks join, the saved state is resharded across the new topology and reloaded. The checkpoint format must support reshard-on-load — DCP does this; older formats often don't. Without it, elasticity reduces to "restart from latest checkpoint after every change," which costs more time than the elasticity saves.

What's the right RAID configuration for local NVMe staging? RAID 0 (no redundancy) is fine for staging — the checkpoint is already replicated to durable tiers, so losing the local copy means re-staging from RAM or just skipping forward. RAID 1 doubles cost for redundancy you don't need. The single decision worth making: stripe across multiple NVMe drives in the node for higher aggregate bandwidth. A node with 4 × 7.68 TB NVMe drives in RAID 0 gets ~30+ GB/s sustained write — enough to absorb most checkpoint shards in seconds.

How do I version checkpoints when the model architecture changes mid-run? You don't, usually — architecture changes are major events that reset the training. If you must (e.g., adding a layer, changing the position-embedding type), embed the architecture version in the checkpoint metadata, write a converter from the old to new format, and validate the converted checkpoint produces the same outputs on a small held-out set before resuming training. The conversion itself is engineering-week-scale work; budget accordingly.

What's the cost of running a continuous checkpoint validator job? Cheap relative to training: a single GPU or even a CPU node can validate sampled checkpoints in the background, since the validation work (checksum verification, occasional forward passes) is far smaller than training itself. Budget ~0.5–1% of cluster compute for continuous validation. Cheap insurance against the catastrophic "we trained for two weeks on corrupted weights and didn't notice" failure mode.

How does checkpoint design interact with confidential computing hardware (H100 CC, Blackwell TEE)? Confidential computing protects data in use; checkpoints are data at rest. The standard pattern: encrypt the checkpoint at the application layer before writing to durable storage, with keys managed by the cluster's KMS. Inside the TEE, the keys are available; outside, the checkpoint is opaque. Performance overhead of AES-256 encryption on checkpoint writes is small (~1–3 GB/s per core with AES-NI), much less than the disk write speed. The complexity is mostly in key management and ensuring the recovery path has access to the keys.

Should I use a separate cluster for the checkpoint validator job? Optionally. The main constraint is the validator needs read access to the checkpoint storage and enough compute to load and forward-pass a sampled checkpoint. For very large checkpoints (multi-TB), the validator needs comparable parallelism to the training job to load efficiently — sharing the cluster (with low-priority scheduling) is cheaper than maintaining a separate one. The pattern of "validate on whatever spare capacity is available" is common.

What's the right way to handle "phantom" checkpoints that look complete but fail to load? A "phantom" — every shard present, sizes look right, but the loader errors or produces NaN forward-pass — is the worst class of corruption. Defenses: per-shard checksums computed at write and verified at load; a background validator that does a forward pass on a sampled prefix of recent checkpoints; explicit _loaded_successfully markers written after a successful load test on the writer-side. Don't trust the existence of a checkpoint as evidence of its validity.

How do I migrate from FSDP1 to FSDP2 without losing in-flight checkpoints? Common pattern: pause training at a clean step, save a consolidated (non-sharded) checkpoint via FSDP1's FULL_STATE_DICT, restart the training script under FSDP2 with --resume_from_consolidated, and have FSDP2's reshard-on-load handle the redistribution. Engineering-week-scale undertaking with a validation step that confirms post-migration loss matches pre-migration on a held-out batch.

What's the right cadence for the validator job vs the training cadence? A 1:10 ratio is a reasonable starting point: if training checkpoints every 15 minutes, the validator samples ~one checkpoint per 2.5 hours. Skewed by validator cost — cheap forward-pass-only validation can run more often; full reload-and-eval validation runs less often. The goal is eventual detection of corruption, not real-time; bad checkpoints have hours of window before they bite recovery.

Can I run multiple training jobs against the same checkpoint storage path? Don't. Each run gets its own path; checkpoint metadata-store rows are immutable once written; retention policy is per-path. The exception is "continue training from prior run" semantics where the new run reads from the prior path read-only and writes to its own new path. Race conditions on shared paths cause subtle corruption that's hard to debug.

What's the operational difference between Lustre and WekaFS in practice? Lustre is HPC-tradition: high theoretical throughput, more operational burden (MDS tuning, OST balance, client-side hangs need attention). WekaFS is purpose-built for AI workloads: easier to operate, often faster on small-file-heavy workloads, but costs more per byte. Frontier-scale teams that already have HPC operators run Lustre; newer AI-native teams typically choose WekaFS or VAST.

How should I think about checkpoint costs vs training compute costs? Rule of thumb: checkpoint infrastructure (storage + network + the small CPU/RAM overhead of staging) is typically 1–3% of total training cost. If yours is meaningfully higher, you're likely over-replicating or under-tiering; if lower, you may be under-investing and paying the recovery tax instead. The right balance is whatever makes expected lost-work per failure roughly equal to checkpoint write overhead.

Are signed checkpoints (cryptographic signatures on weights) becoming standard? For published / partner-handoff checkpoints, yes — by 2026 several model registries require Sigstore-style signatures on published artifacts to bind weights to a provenance chain. For internal training checkpoints, less common: the audit log + bucket RBAC story is typically considered sufficient. Signed checkpoints will likely become a regulatory requirement under the EU AI Act's general-purpose-AI obligations.

How does the cluster scheduler interact with checkpoint design? Schedulers (Kubernetes + Volcano, Slurm, internal NVIDIA Base Command, Google's Borg) need to know when a job is "safe to evict" — i.e., when its state is checkpointed. The integration: training job reports its last-good-checkpoint step to the scheduler; scheduler can preempt with bounded lost work. Without this integration, preemption is either disabled (capacity wasted) or unsafe (work lost).

What's the impact of checkpoint design on the "training-data lineage" compliance question? Each checkpoint metadata row should record the data shard cursor (which examples have been consumed up to this step). On regulatory request — "what data influenced this model?" — the answer derives from the metadata: enumerate all data shards consumed between training start and the published checkpoint's step. Without this lineage, the compliance answer is "we don't know." With it, the answer is a reproducible SQL query against the metadata catalog.

Is there a single best checkpoint cadence number for a 10k-GPU run? Roughly every 15 minutes is the consensus 2026 default — failure rate at 10k GPUs is on the order of every few hours, async checkpoint write is on the order of 10–30 seconds, and the cadence equation (lost-work ≈ write-overhead) lands in the 10–30 minute range. Calibrate against your actual failure rate and write speed.

How do erasure-coded checkpoints recover when more than m nodes fail simultaneously? They don't — that's the point of m. The fallback is the next-tier-down checkpoint (cluster FS → object store). Pick m such that simultaneous failures of m+1 nodes is rare enough to accept the tier-down recovery cost. For RS(8,2), losing 3 nodes simultaneously means falling back to cluster-FS-stored checkpoint; happens rarely enough at sub-100k-GPU scale to be tolerable.

What's the right way to test the recovery path before production? Chaos engineering. Run a "kill a random rank every hour" experiment on a non-critical training run and measure: (1) does the supervisor detect the failure? (2) does recovery complete within SLO? (3) does the training loss continue smoothly from the recovery point? Repeat with progressively worse failures — rack-level, switch-level, region-level. The first time you run this, expect bugs; the goal is to find them before a production run does.

Does training-data versioning need to be checkpoint-aligned? Strongly recommended. A checkpoint should reference the training-data version (hash, version ID, or cursor position) by metadata. On resume, the framework verifies the data version matches; if it doesn't (someone updated the dataset), the resume is rejected. Prevents the silent failure where a "resumed" run actually saw different data than the prior run.

What's the role of object-versioning on the bucket for checkpoint storage? Defense in depth. With object-versioning enabled, an accidental overwrite or deletion leaves the prior version recoverable. Pair with MFA-delete on the bucket policy to prevent automated deletion. The cost is modest (storage of old versions, ideally with lifecycle to cool/archive after N days). The benefit is that "we accidentally overwrote the only good checkpoint" is recoverable instead of fatal.

How do I handle checkpoint compatibility across PyTorch versions? DCP and safetensors are forward-compatible across PyTorch minor versions; pickle-format .pt files are not. Pin the PyTorch version per training run; record it in the checkpoint metadata. On resume, verify the resume environment matches; if it doesn't, the load may succeed silently with subtle numerical differences (different default dtypes, different op implementations). Pinning prevents this entire class of bug.

Is there a future for "differential checkpointing" — only storing the diff since last checkpoint? Active research area; not yet standard production. The challenge is that optimizer state (momentum, variance) is dense and changes on every step, so the diff is approximately the same size as the full state for the dominant cost term. For weight-only checkpoints with relatively few updates between checkpoints (e.g., LoRA fine-tuning, slow-learning-rate phases), differential checkpointing can save substantial space. For pretraining, the savings are marginal.

---

Glossary

• Async checkpoint — copies state to staging buffer, then writes in background.
• Atomic finalization — write to temp + rename so partial writes aren't visible.
• Consolidated checkpoint — single coherent file, topology-independent.
• Cadence — how often checkpoints are taken.
• Optimizer state — momentum/variance buffers and master weights.
• Parallel filesystem — distributed storage striped across many nodes (Lustre, GPFS, etc.).
• Recovery time — time to load and resume from a checkpoint.
• Sharded checkpoint — per-rank checkpoint files, fast to write but topology-bound.
• Staging buffer — host-memory area for tensor data before durable write.
• Tiered storage — multiple storage layers with different speed/durability trade-offs.

---

References

• Bamboo — Thorpe et al., 2023. "Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs." NSDI 2023. arXiv:2204.12013. Preemption-aware training infra.
• Check-N-Run — Eisenman et al., 2022. "Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models." NSDI 2022. arXiv:2010.08679.
• ZeRO — Rajbhandari et al., 2019. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." arXiv:1910.02054. DeepSpeed's foundational paper, includes checkpoint strategies.
• safetensors — Hugging Face's safer-than-pickle file format. github.com/huggingface/safetensors.
• PyTorch Distributed Checkpoint — see torch.distributed.checkpoint docs.
• Megatron-LM — Narayanan et al., 2021. arXiv:2104.04473. Checkpoint utilities in source.
• Mooncake — Qin et al., 2024. arXiv:2407.00079. Related work on distributed KV/state storage with overlapping concerns.
• Lustre, GPFS, WekaFS — vendor technical documentation for the parallel filesystems used in production training.
• ZeRO-Infinity — Rajbhandari et al., 2021. "ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning." arXiv:2104.07857. Covers checkpointing patterns for very large models with hierarchical offload.
• PyTorch Distributed Checkpoint (DCP) — official docs. pytorch.org/docs/stable/distributed.checkpoint.html.
• The Tail at Scale — Dean & Barroso, CACM 2013. research.google/pubs/the-tail-at-scale/. Foundational analysis of how failure and latency compose at scale.
• DeepSeek-V3 Technical Report — DeepSeek-AI, 2024. arXiv:2412.19437. Candid engineering details on FP8 training stability, recovery, and the parallelism layout's role in localizing failures.
• FSDP — Zhao et al., 2023. "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel." arXiv:2304.11277. The sharding model that DCP integrates with.