Mixed Precision LLM Training: The Complete Guide

Q: Is FP8 stable for training across all model sizes?

Smaller models (< 1B): can be tricky. Medium (10-100B): generally stable with care. Large (100B+): sometimes more stable due to averaging.

Q: How do I validate FP8 training quality?

Compare loss curves with BF16 baseline. Eval downstream tasks. Monitor activation distributions.

Q: Should I use e4m3 or e5m2 for forward pass?

e4m3 (more precision, less range). e5m2 typically used for backward pass (gradients).

Q: What's the hardware support for FP8?

H100, H200, B100, B200, Blackwell — all support FP8 natively. Older GPUs (A100): no.

Q: Will FP4 replace FP8 for training?

Likely not in the near term. FP4 is harder to train. FP8 is the current sweet spot.

Q: How does mixed precision affect convergence?

Properly done: minimal impact. Improperly done: divergence.

Q: Should I tune learning rate when switching precisions?

Sometimes. Test empirically. Often the same hyperparameters work.

Q: What about gradient clipping with mixed precision?

Yes — gradient clipping is precision-independent. Clip values before scaling.

Q: How does mixed precision interact with gradient accumulation?

Accumulator should be FP32 to avoid loss of precision over many micro-batches.

Mixed precision training is the practice of using lower-precision formats (BF16, FP8, FP4) for the bulk of forward and backward passes while keeping critical pieces (optimizer state, master weights) in higher precision. The original recipe (Micikevicius et al., arXiv:1710.03740) defined the pattern; FP8 followed in 2022 (Micikevicius et al., arXiv:2209.05433). Done well, it doubles or quadruples training throughput at near-zero quality cost. Done badly, it produces models that fit in memory but don't converge — closely related to the failure modes in quantization tradeoffs at inference time.

Key takeaways
Mental model: mixed precision in one minute
The precision formats
Why mixed, not pure
Loss scaling and the FP16 era
BF16: the safe default
FP8: the modern frontier
FP4: emerging
NVIDIA Transformer Engine
Framework support
Auditing a mixed-precision run
Common failure modes
Worked example: switching a training run to FP8
Per-format throughput math
When mixed precision breaks
Comparing FP8 implementations
FP4 training in production
Mixed precision and distributed parallelism
Per-precision deep dive: every format you might use
Transformer Engine internals: scaling, recipes, and gotchas
DeepSeek FP8 training recipe in detail
MS-AMP and torchao FP8: the other implementations
FP8 with FSDP, TP, and pipeline parallelism
Per-model-size feasibility: where FP8 pays
MoE and FP8: the special case
Fine-tuning in FP8 vs BF16
Learning-rate schedules and precision interactions
FP8 and gradient accumulation
Communication precision in detail: BF16 vs FP32 all-reduce
Numerical-failure taxonomy: diagnosing FP8 runs
Checkpoint precision: what to save and reload
torch.compile and FP8: the interaction
Worked example: budget for a 70B FP8 vs BF16 run
The bottom line
FAQ
Extended FAQ
Glossary
References

Key takeaways

The defaults that work in 2026:

BF16: the safe default for most training. No loss scaling needed. ~2× faster than FP32 on Hopper+.
FP8 (e4m3 forward, e5m2 backward): standard for frontier training. ~2× faster than BF16 on H100+. Quality cost ~0.1 points with proper calibration.
FP4: emerging on Blackwell. ~2× faster than FP8. Quality cost still being characterized.
FP32 master weights + Adam state: required regardless of forward/backward precision. Gives optimizer the precision it needs.

The non-obvious thing: lower precision is not free. Each step down the precision ladder requires more careful handling — calibration, loss scaling, layer-specific exclusions, gradient clipping. The throughput wins compound; so do the failure modes.

Mental model: mixed precision in one minute

The named problem is the numerical-range cliff: every step down the precision ladder shrinks the dynamic range of representable numbers, and gradients are exactly the values that live near the bottom of that range. FP32 covers ~10^-38 to ~10^38, so almost nothing underflows. FP16 covers only ~6e-5 to 65504, so small gradients flush to zero mid-training and the model silently stops learning. FP8 e4m3 has only ~448 of usable range and the cliff gets steeper still. Yet FP32 throughput is 4-16x slower than FP8 on Hopper-class tensor cores. You cannot afford to stay in FP32, and you cannot survive a naive drop to FP8.

The core idea is to use different precisions in different places, matched to what each value actually needs. Forward activations and weights tolerate low precision because they're bounded; gradients need wide range; the optimizer needs high mantissa precision to accumulate tiny updates over many steps. So the modern recipe runs matmuls in FP8 (e4m3 forward, e5m2 backward), keeps activations in BF16 between matmuls, keeps a master copy of weights in FP32, and runs Adam moments in FP32. Per-tensor scaling factors are calibrated on the fly so each FP8 tensor uses its full range — when an outlier blows past the max, the scale adapts before the next step.

Aspect	Pure FP32	BF16 mixed	FP8 mixed (e4m3/e5m2)
Matmul precision	FP32	BF16	FP8
Master weights	FP32	FP32	FP32
Optimizer state	FP32	FP32	FP32
Throughput vs FP32	1x	~2x	~4x (Hopper), ~8x (Blackwell w/ FP4)
Memory per param	4 bytes	2 bytes	1 byte (plus FP32 master)
Failure mode	None	Rare loss spikes	Underflow, outlier blow-ups
When it pays off	Debugging only	Always	1B+ params, Hopper+

Conceptually:

# Naive: cast everything to FP8 and pray
model = model.to(torch.float8_e4m3fn)  # this will not converge

# Real recipe (Transformer Engine handles all of it):
from transformer_engine.pytorch import fp8_autocast, DelayedScaling
recipe = DelayedScaling(margin=0, fp8_format=Format.HYBRID)
with fp8_autocast(enabled=True, fp8_recipe=recipe):
    loss = model(x)              # matmuls in FP8, activations BF16
loss.backward()                   # gradients accumulated in FP32 master
optimizer.step()                  # Adam state in FP32

One number to remember: a properly calibrated FP8 mixed-precision run delivers ~2x the throughput of BF16 with quality loss under ~0.1 points on standard benchmarks (Micikevicius et al., 2022; Llama-3, DeepSeek-V3 production). Without per-tensor scaling, the same setup diverges within thousands of steps.

The rest of this guide is everything that extends or depends on that idea — what each format actually represents, when to exclude layers from FP8, how Transformer Engine implements the recipe, and how to debug a run that suddenly NaNs at step 50,000.

The precision formats

Format	Bits	Exponent	Mantissa	Dynamic range	Mantissa precision
FP32	32	8	23	~10⁻³⁸ to ~10³⁸	~7 decimal digits
TF32	19 (32-bit storage)	8	10	~10⁻³⁸ to ~10³⁸	~3 decimal digits
FP16	16	5	10	~6×10⁻⁵ to ~6×10⁴	~3 decimal digits
BF16	16	8	7	~10⁻³⁸ to ~10³⁸	~2 decimal digits
FP8 e4m3	8	4	3	~2⁻⁹ to ~448	~0.5 decimal digits
FP8 e5m2	8	5	2	~2⁻¹⁶ to ~57344	~0.4 decimal digits
FP4 e2m1	4	2	1	~0.125 to 6	very coarse

Two key observations:

Dynamic range (set by exponent bits): how big and small can numbers be? Important for gradients (small) and activations (sometimes very large).
Mantissa precision (set by mantissa bits): how finely can you represent values within the dynamic range? Important for accumulating many small numbers without losing them in noise.

The art of mixed-precision training is choosing the right format for each operation based on its dynamic range and precision requirements.

Why mixed, not pure

You can't train pure FP8 or pure BF16 because:

Optimizer state needs FP32 precision. Adam moments accumulate over millions of steps; FP16/BF16 lose precision and the optimizer drifts.
Master weights need FP32. Gradient updates are often very small; if master weights are BF16, those updates round to zero.
Loss accumulation across micro-batches needs higher precision. Sum of many small gradients accumulates rounding errors in lower precision.

The standard pattern:

Forward pass: BF16 or FP8.
Backward pass: BF16 or FP8.
Master weights: FP32.
Optimizer state (Adam m, v): FP32.

Memory cost per parameter:

Pure FP32: 16 bytes (weight + grad + Adam m + Adam v).
Mixed BF16: 12 bytes (BF16 weight + BF16 grad + FP32 master + FP32 Adam).
Mixed FP8: 10 bytes (FP8 weight + FP8 grad + FP32 master + FP32 Adam).

So FP8 saves 6 bytes/param vs FP32 — a 37% memory reduction. Plus 4× tensor core throughput.

Loss scaling and the FP16 era

A historical note. FP16 was the first widely-deployed mixed-precision format (around 2018-2020). It had a major problem: dynamic range too narrow.

In FP16, anything below ~6×10⁻⁵ underflows to zero. Gradients are often this small. Half-precision gradients literally vanish.

The fix: loss scaling, introduced in the original mixed-precision paper (Micikevicius et al., arXiv:1710.03740). Multiply the loss by a large factor (1024 or higher) before backward pass. Gradients are scaled up by the same factor. Divide gradients by the factor before applying to FP32 master weights.

loss = compute_loss()
scaled_loss = loss * scale_factor    # e.g., 1024
scaled_loss.backward()
# Now gradients are scaled up by scale_factor
for param in model.parameters():
    param.grad = param.grad / scale_factor
optimizer.step(param.grad in FP32)

Dynamic loss scaling: scale_factor adapts. Increase it on success, halve it on Inf/NaN gradients.

This is why FP16 training requires more care than BF16. BF16 has FP32-equivalent dynamic range and doesn't need loss scaling at all.

BF16: the safe default

BF16 became the standard for transformer training because it has FP32's dynamic range with half the bits. No loss scaling needed; the format just works.

When BF16 is right

Modern transformer training (Llama, Mistral, Qwen, etc.).
Any setup where you don't need FP8's speedup specifically.
Stable training on Hopper, Ampere, and modern AMD.

When BF16 isn't enough

Frontier-scale training where FP8's 2× speedup matters for cost.
Memory-tight setups where the 25% additional savings of FP8 are needed.
Very long training runs where every percent of throughput matters.

Configuring BF16

PyTorch:

from torch.cuda.amp import autocast
model = model.to(torch.bfloat16)
# or use autocast for automatic casting:
with autocast(dtype=torch.bfloat16):
    loss = model(input)

Megatron-LM (Shoeybi et al., arXiv:1909.08053), DeepSpeed/ZeRO (Rajbhandari et al., arXiv:1910.02054), and FSDP all support BF16 via flags — see the broader distributed LLM training guide for how this composes with NCCL collectives and training networking.

FP8: the modern frontier

FP8 is the current frontier of training precision. Hopper (H100) introduced FP8 tensor cores (NVIDIA H100 whitepaper); Blackwell (B200) extended them. Production FP8 pretraining at frontier scale was demonstrated end-to-end by DeepSeek-V3 (DeepSeek-AI, arXiv:2412.19437), building on the FP8-LM / MS-AMP recipe (Peng et al., arXiv:2310.18313). See also our notes on NVIDIA datacenter GPUs and the LLM serving implications of FP8 weights.

Two FP8 formats

e4m3 (4 exponent, 3 mantissa, 1 sign): finer precision, narrower dynamic range. Good for activations and forward pass.
e5m2 (5 exponent, 2 mantissa, 1 sign): wider dynamic range, coarser precision. Good for gradients during backward pass.

The asymmetry matches the actual values:

Activations: values typically in moderate range, want precision.
Gradients: wider dynamic range due to gradient magnitudes varying across layers, want range over precision.

Calibration and scaling

FP8 needs per-tensor scaling factors to map the float range to the FP8 range. Without proper scales:

Outliers overflow → garbage gradients.
Most values cluster near zero → effective bit width drops.

Modern frameworks compute scales adaptively:

Per-tensor scaling: one scale per tensor, computed dynamically based on observed maxima.
Per-channel scaling: separate scales per output channel. More accurate, more memory.

The "delayed scaling" technique (Transformer Engine) tracks scales over a window and updates them, smoothing out transient outliers.

When FP8 wins

Frontier training runs (saves real money on long runs).
Memory-constrained training where the 25% memory savings matter.

When FP8 hurts

Smaller training runs where the throughput gain doesn't justify the operational complexity.
Models with known numerical sensitivity (some attention configurations).

Layer-specific FP8

Frontier training in 2026 often uses FP8 for MLP and attention matmuls but keeps:

LayerNorm and softmax in BF16 (numerical sensitivity).
Loss computation in FP32.
Final logits in BF16 or FP32.

Transformer Engine handles this layer-specific routing automatically.

FP8 quality cost

Empirical: ~0.1 points on standard benchmarks, properly calibrated. Sometimes 0.2 points. Trivial relative to the throughput gain.

If you see > 0.5 points loss vs BF16 baseline, calibration is misconfigured. Investigate before scaling up.

FP4: emerging

Blackwell introduced FP4 tensor cores. e2m1 format. Throughput on B200: 2× FP8.

FP4 training is bleeding-edge in 2026:

Some research training has used FP4 forward + FP8 backward.
Quality data is preliminary; ~0.5–1 point cost vs BF16 typical.
Frameworks support is partial (Transformer Engine has experimental FP4).

By 2027, expect FP4 to become standard for forward-pass MLPs in frontier training, similar to how FP8 became standard 2024-2026.

For now: FP4 if you're at frontier scale and willing to invest in figuring out the quality-throughput tradeoff. BF16 or FP8 otherwise.

NVIDIA Transformer Engine

Transformer Engine (TE) is NVIDIA's library for FP8/FP4 training. It provides:

FP8/FP4-aware modules (Linear, LayerNorm, MultiHeadAttention).
Automatic per-tensor scaling.
Layer-specific precision routing (some layers in FP8, others in BF16).
Recipe-based configuration.

import transformer_engine.pytorch as te

# Replace standard Linear with TE Linear
linear = te.Linear(hidden_size, hidden_size, params_dtype=torch.bfloat16)

# Use FP8 recipe
fp8_recipe = te.recipe.DelayedScaling(
    fp8_format=te.Format.HYBRID,  # e4m3 forward, e5m2 backward
    amax_history_len=16,
    amax_compute_algo="max",
)
with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
    output = linear(input)

TE is required for FP8 training in production. Pure-PyTorch FP8 is more brittle.

Framework support

Framework	BF16	FP8	FP4
PyTorch native	✅	⚠ via TE	⚠ via TE experimental
Megatron-LM	✅	✅ via TE	⚠ early
DeepSpeed	✅	✅	❌
NeMo	✅	✅	⚠ early
JAX	✅	✅ via libraries	⚠ early
Lightning	✅	✅ via TE	❌

NVIDIA's stack (Megatron, NeMo, TE) has the deepest FP8/FP4 support. Open ecosystem (PyTorch native, DeepSpeed) follows close behind.

Auditing a mixed-precision run

Things to check periodically during training:

Loss curve

Should be smooth and monotonically decreasing (with some noise). Sharp spikes or divergence = numerical instability.

Gradient norms

Track L2 norm of gradients. Should be O(1) typically. Sudden spikes or zeros indicate FP8 overflow/underflow.

Activation statistics

Track mean and standard deviation of intermediate activations. Drifting means = numerical drift. Very large maxes = upcoming overflow.

Loss scaling history

For FP16, watch the loss scale value. If it's halving frequently, gradient overflow is happening too often.

For FP8, watch the per-tensor scaling factors. Stable scales across iterations = healthy.

Compare to BF16 baseline

Periodically validate by running a few iterations in BF16 and comparing loss values. Should be within ~1% of FP8.

Common failure modes

NaN/Inf in gradients

Cause: FP8 overflow or unscaled FP16. Loss scaling is misconfigured.

Fix: increase loss scaling, check for outlier inputs, verify TE recipe.

Loss diverges after ~10000 steps

Cause: numerical drift accumulating. Common with aggressive FP8 settings.

Fix: switch the suspect layer (often attention or layer norm) to BF16. Reduce learning rate slightly.

Quality regression vs BF16 baseline

Cause: insufficient calibration set, wrong FP8 format choice, or numerical issues.

Fix: run for longer with FP8, expand calibration set. If gap doesn't close, fall back to BF16 for the troubled layers.

Slower than expected throughput

Cause: FP8 enabled but per-step time hasn't dropped. Often: model has too few FP8 ops to amortize overhead.

Fix: profile with NVIDIA Nsight Systems. Check that FP8 GEMMs are actually running (not falling back to BF16 due to misconfig).

Bad scaling on a specific layer

Cause: that layer's activations have unusual distribution.

Fix: exclude from FP8, use BF16 for that layer.

Worked example: switching a training run to FP8

You have a Llama-3 8B training run in BF16. You want to switch to FP8 for ~2× throughput.

Step 1: baseline

Run 1000 steps in BF16. Note final loss, gradient norms, throughput. This is your reference.

Step 2: enable FP8 with default recipe

from transformer_engine.common.recipe import DelayedScaling, Format

fp8_recipe = DelayedScaling(
    fp8_format=Format.HYBRID,  # e4m3 fwd, e5m2 bwd
    amax_history_len=16,
)

Run another 1000 steps. Compare loss to BF16 baseline.

Step 3: validate

If FP8 loss is within 0.5% of BF16 loss, you're good.

If FP8 loss is significantly higher:

Check NaN/Inf rate. >0% = numerical issues.
Check per-tensor scales. Wide variation = poor calibration.
Try different recipe (e.g., longer amax history).

Step 4: scale up

Confirm throughput is ~2× BF16. If it isn't, profile.

Step 5: long-run validation

Train for 100k steps. Compare loss curve to BF16 reference at the same step counts. They should track within 1-2%.

If loss drifts: identify which layer is responsible (track per-layer activation norms), exclude from FP8.

A short history of mixed-precision training

Mixed precision wasn't always standard. A timeline:

2014-2015: pure FP32 was the default. Single-precision arithmetic. Slow but stable.

2017: Micikevicius et al. publish "Mixed Precision Training" — the foundational paper introducing FP16 + FP32 master weights + loss scaling. NVIDIA's Apex library popularized it.

2018-2019: FP16 mixed-precision becomes mainstream for transformer training. NVIDIA's Tensor Cores on V100 made it 2× faster than FP32.

2020: BF16 (Brain Float 16) introduced by Google (TPU) and adopted by NVIDIA (A100). Solves FP16's narrow-dynamic-range problem. Becomes the default for transformer training.

2022: Hopper (H100) launches with FP8 tensor cores. NVIDIA's Transformer Engine library makes FP8 training practical.

2023-2024: FP8 becomes standard for frontier-scale training. Llama-3, DeepSeek-V2, etc. trained with FP8.

2024: Blackwell (B200) launches with FP4 tensor cores. Some research training uses FP4 for forward pass.

2025-2026: FP8 is the production default for new training runs on Hopper+. FP4 is emerging.

Future (2027+): FP4 likely becomes standard for forward pass. New formats (Microsoft's MX, OCP's MXFP6, etc.) may proliferate.

The pattern: each generation adds new lower-precision formats while keeping previous ones. Mixed precision now means choosing the right format for each operation.

Per-format quality benchmarks

Across the major formats, typical quality cost vs FP32 baseline (Llama-3 70B-class, MMLU benchmark):

Format	MMLU drop	RULER 32k drop	Notes
FP32	baseline	baseline	reference
TF32	<0.05	<0.05	essentially free
FP16 (with loss scaling)	0.1	0.3	some instability
BF16	0.05	0.1	safe default
FP8 e4m3 (forward), e5m2 (backward)	0.1-0.2	0.5-1.0	well-calibrated
FP8 e4m3 (everything)	0.3-0.5	1.0-2.0	not recommended
INT8 mixed	0.2-0.4	0.8-1.5	for inference deployment
FP4 e2m1	0.5-1.0	1.0-3.0	early data

Quality numbers improve over time as calibration techniques mature. FP8 numbers from 2023 papers were worse than current; expect FP4 to follow the same trajectory.

Numerical stability deep dive

Mixed-precision training has specific failure modes. Understanding them helps tune.

Underflow

Values smaller than the format's minimum representable number become zero.

For FP16: anything below ~6×10⁻⁵ underflows. Gradients are often this small. This is why FP16 needs loss scaling.

For BF16: anything below ~10⁻³⁸ underflows (basically same as FP32). Doesn't happen in practice.

For FP8 e4m3: anything below ~2⁻⁹ ≈ 0.002 underflows. Most activations are above this; gradients (in e5m2) have wider range, so OK.

Overflow

Values larger than the format's maximum representable number become infinity.

For FP16: anything above ~6×10⁴ overflows. Loss scaling pushes large gradients into this range; needs dynamic adjustment.

For BF16: anything above ~10³⁸ (basically FP32 max). Doesn't happen.

For FP8: e4m3 max is ~448. Activations need careful scaling. Per-tensor or per-channel scales handle this.

Loss spikes

Sudden jumps in loss during training. Often caused by:

Gradient overflow → divergence on a single bad batch.
Numerical drift → accumulated errors over many steps.
Bad data → outlier batch with extreme gradients.

Mitigations: gradient clipping (typical 1.0), dynamic loss scaling, more aggressive calibration.

NaN propagation

Once a NaN appears, all subsequent operations propagate it. Detect early:

if torch.isnan(loss).any():
    raise ValueError("NaN detected — investigate")

Common causes:

FP8 overflow.
Division by zero (rare in attention).
Bad input data.

Stop and investigate when NaN appears. Don't just skip and continue.

Calibration deep dive

For FP8 (and lower), calibration determines per-tensor scaling factors. Poor calibration kills quality.

Static calibration

Pre-compute scales using a calibration dataset:

from transformer_engine.pytorch.fp8 import fp8_autocast

# Run calibration pass
with fp8_autocast(enabled=False):  # collect statistics in BF16
    for batch in calibration_set:
        model(batch)

# Now run training with computed scales
with fp8_autocast(enabled=True):
    for batch in training_set:
        loss = model(batch)
        loss.backward()

Static calibration is fast (one pass) but may not represent training-time activations.

Dynamic calibration (delayed scaling)

Track activation maxima over a sliding window during training. Update scales each iteration.

NVIDIA Transformer Engine's default is dynamic with a 16-step history.

fp8_recipe = DelayedScaling(
    margin=0,
    fp8_format=Format.HYBRID,
    amax_history_len=16,
    amax_compute_algo="max",  # or "most_recent"
)

This is what most production FP8 training uses.

Per-channel vs per-tensor

Per-tensor: one scale per tensor. Cheap but misses per-channel variations.

Per-channel: separate scales per output channel. ~10% memory overhead. Recovers ~50% of the per-tensor quality gap.

For weights: per-channel scaling is standard. For activations: per-tensor or per-channel depending on format. For KV cache: per-channel for K, per-token for V (KIVI).

Block-wise calibration (FP4)

For FP4, even per-channel isn't enough. Block-wise scaling (one scale per N elements within a tensor) is the modern approach.

NVIDIA's MXFP4 format uses block-wise scaling. Microsoft's MX formats are similar.

Calibration data choice

Random samples from training data: best representativeness.
Synthetic data: faster but less accurate.
Production traces: ideal if you have real production data.

Don't over-think calibration data. 100-1000 representative batches is enough.

Mixed precision in distributed training

When you combine mixed precision with TP/PP/DP, additional considerations.

TP and FP8 communication

TP all-reduces happen in BF16 even when forward/backward use FP8. Why: collective operations need higher precision to avoid accumulating quantization errors across many GPUs.

Concrete: a TP=8 all-reduce of FP8-quantized activations would suffer significant quality loss. Transformer Engine handles the BF16 promotion transparently.

FSDP and FP8 weights

FSDP sharding works fine with FP8. Each rank stores its shard of FP8 weights. All-gather happens at FP8 (or BF16, depending on framework).

The optimizer state (Adam m and v) stays FP32 regardless. Memory savings from FP8 weights are real but smaller than they appear because optimizer state dominates memory.

Gradient accumulation in mixed precision

Accumulate gradients in FP32, then quantize for the optimizer step:

for micro_batch in micro_batches:
    with autocast(dtype=torch.bfloat16):
        loss = model(micro_batch)
    loss.backward()  # accumulates in FP32 master grads

optimizer.step()  # FP32 master weights are updated; FP8 weights resync

Most frameworks handle this automatically.

Loss aggregation across ranks

When computing average loss for logging, use FP32:

loss_value = loss.detach().float()
dist.all_reduce(loss_value, op=dist.ReduceOp.AVG)

Don't average BF16 losses across many ranks — accumulated rounding error is real.

Per-precision deep dive: every format you might use

The format inventory has expanded faster than the docs. A complete per-format reference for the precisions that matter in 2026.

FP32 (IEEE 754 single-precision)

The reference precision. 32 bits, 8-bit exponent, 23-bit mantissa. Dynamic range ~10^-38 to ~10^38, mantissa precision around 7 decimal digits. Used today for: optimizer state (Adam m and v), master weights, loss accumulation across micro-batches, gradient norm reduction. Almost no production training run uses FP32 for the hot matmul path — the throughput cost is too high.

TF32 (NVIDIA TensorFloat-32)

NVIDIA's hardware compromise: 32-bit storage, but only 10-bit mantissa during tensor-core matmul. Introduced on Ampere (A100). Same dynamic range as FP32, much less precision. Treated by frameworks as a transparent acceleration for FP32 matmuls; the user doesn't have to think about it. Useful as a fallback when BF16 has a stability issue, otherwise mostly obsolete on Hopper+.

BF16 (bfloat16)

16 bits, 8-bit exponent, 7-bit mantissa. Same dynamic range as FP32, much less mantissa precision. The de facto safe default for transformer training in 2026. No loss scaling required. Hopper, Ampere, AMD MI300X, TPU v4/v5/v6 all support BF16 tensor cores at high throughput. Most published recipes (Llama 3, Llama 3.3, Qwen 2.5/3) use BF16 for the entire training run.

FP16 (IEEE 754 half-precision)

16 bits, 5-bit exponent, 10-bit mantissa. More mantissa precision than BF16 but much narrower dynamic range (6e-5 to 65504). Underflow on gradients is the structural failure mode. Required loss scaling to be usable. Largely supplanted by BF16 for training; still relevant for inference where the higher mantissa precision helps.

FP8 E4M3

8 bits, 4-bit exponent, 3-bit mantissa. Range ~2^-9 to ~448. The forward-pass format in the modern FP8 recipe. The 3-bit mantissa gives finer precision than E5M2, useful for activations and weights. Used in Transformer Engine's HYBRID recipe for forward matmuls and weight storage.

FP8 E5M2

8 bits, 5-bit exponent, 2-bit mantissa. Range ~2^-16 to ~57344. The backward-pass format in the modern FP8 recipe. The 5-bit exponent gives wider dynamic range than E4M3, useful for gradients. Used in Transformer Engine's HYBRID recipe for backward matmuls.

FP4 NVFP4

NVIDIA's flavor of FP4: 4 bits, with a microscaling factor per block of 16 elements. Introduced on Blackwell (B200). Each 16-element block has its own E4M3 scaling factor that brings the per-block dynamic range up to E4M3 levels while keeping the per-element representation at FP4. Demonstrated in research for inference and some forward-pass training; full-pipeline FP4 training is still experimental.

FP4 MXFP4

A standardized (Open Compute Project, OCP) microscaling FP4. Similar idea to NVFP4 — per-block scaling factor — with slightly different format choices (different shared scale type, different block size in some implementations). Supported on Blackwell and on some AMD MI300-series silicon. The standardization matters because it allows cross-vendor portability for the small but growing class of FP4 workloads.

INT8

8-bit signed integer. Not a floating-point format. Used primarily for inference quantization (post-training quantization, weight-only quantization) where the model is calibrated to use INT8 weights and/or activations. Less relevant for training because integer arithmetic doesn't compose with gradient computation as naturally as floating-point.

INT4 and lower

4-bit weight-only quantization (GPTQ, AWQ, EXL2) is the inference floor in 2026 for many production deployments. Training in INT4 is not viable; the inference use case is covered in quantization tradeoffs.

The format choice matrix

Use case	Recommended primary format	When to escalate
Master weights	FP32	Never escalate; always FP32
Optimizer state (Adam m, v)	FP32	Never escalate; always FP32
Forward matmul	FP8 E4M3 (Hopper+) or BF16	BF16 if FP8 destabilizes
Backward matmul	FP8 E5M2 (Hopper+) or BF16	BF16 if FP8 gradients NaN
Activations between matmuls	BF16	FP32 for the sensitive LM head
LayerNorm	BF16 or FP32	FP32 if loss spikes
Softmax (attention)	BF16 or FP32	FP32 for long-context stability
Loss accumulation	FP32	Never escalate; always FP32
Gradient all-reduce	BF16 or FP32	FP32 for very large worlds

The pattern is consistent: anything that accumulates (master weights, optimizer state, loss, large gradient reductions) stays in FP32. Anything that's a one-shot dense operation (matmuls, layer activations) drops to the lowest precision the layer tolerates.

Transformer Engine internals: scaling, recipes, and gotchas

NVIDIA Transformer Engine is the production reference implementation for FP8 training on Hopper and Blackwell. Understanding what it does internally is the difference between using it competently and being confused when it misbehaves.

Per-tensor scaling

The FP8 format has a narrow dynamic range; arbitrary tensor values do not fit naively. The fix is to multiply each tensor by a per-tensor scale factor before quantizing to FP8, then multiply the result of the matmul by the inverse scale factor. Each scale factor is chosen so that the tensor's max absolute value lands at the top of the FP8 range, using all 8 bits of precision.

Transformer Engine maintains a scale factor per (tensor, format) pair: separate scales for forward activations, forward weights, backward gradients. The scales are updated continuously based on observed tensor magnitudes.

Delayed scaling (the default recipe)

The naive approach — measure the tensor's max, set scale, quantize, run matmul — has a synchronization problem. Computing max on the GPU and then using it for quantization requires a kernel boundary that breaks the matmul fusion.

Delayed scaling solves this with a small lag: the scale used for tensor X at step t is the scale that was right for tensor X at step t-1 (or an exponentially-weighted average of recent steps). The scale is computed asynchronously in the background and applied one step late. The accuracy cost is minimal — tensor distributions change slowly across consecutive training steps — and the throughput benefit is large.

History-based scaling

A refinement: the scale at step t is set based on the max of tensor X across the last N steps (typically N = 16 or 32), not just the previous step. Smooths over single-step outliers that might otherwise drive the scale up and waste precision. Transformer Engine's default DelayedScaling recipe uses this.

Per-row vs per-tensor scaling

The basic version uses one scale per tensor. A finer version (used in torchao's FP8 implementation and several research recipes) uses one scale per row of the weight matrix. The benefit is that outlier columns or outlier rows don't drag down the scale of the entire tensor. The cost is more scale factors to track.

DeepSeek-V3's recipe ([arXiv:2412.19437]) uses a finer block-wise scaling — one scale per 128x128 block of the weight matrix. The block scheme captures local outlier patterns while keeping the number of scale factors manageable.

The HYBRID recipe

Transformer Engine's default for transformer training: forward matmul uses E4M3 weights and E4M3 activations; backward matmul uses E5M2 gradients. The asymmetry matches the value distributions — forward values are bounded and want precision; gradients want wider dynamic range.

Skipping the BAcc accumulator promotion

A subtle stability trick: Transformer Engine accumulates FP8 matmul partial products in FP32 (the BAcc, "block accumulator," accumulator). This is the default; explicitly skipping the FP32 accumulation (running the matmul in pure FP8) breaks stability immediately. The lesson: FP8 is the multiplicand precision, not the accumulator precision. Every reliable FP8 recipe accumulates in higher precision.

FP16 master weights vs FP32 master weights

Most recipes use FP32 master weights and explicitly tolerate the FP32 memory cost. Some experimental recipes use FP16 master weights with periodic re-anchoring to a slower FP32 checkpoint. The memory savings (4 bytes per parameter dropped to 2) are real but the additional complexity is significant. Production stacks generally stay with FP32 master weights.

When Transformer Engine fights you

Common patterns:

NaN appears mid-training. Usually a scale factor that hasn't adapted to a new outlier. Fix: shorter history window, more aggressive scale floor.
Loss diverges at step 50K. Usually a layer whose distribution has drifted out of the FP8 range. Fix: exclude that layer from FP8.
Throughput is worse than BF16. Usually a layer that is too small to amortize the FP8 quantization overhead. Fix: keep small layers in BF16.

The general operational pattern: instrument the run with per-layer scale factor logging, and treat any scale factor that has been at its max for many steps as a candidate for exclusion from FP8.

DeepSeek FP8 training recipe in detail

DeepSeek-V3 is the most thoroughly documented frontier-scale FP8 training run as of 2026. Its published recipe ([arXiv:2412.19437]) deserves detailed analysis because it codifies what frontier FP8 actually requires.

Block-wise scaling at 128 granularity

DeepSeek-V3's main innovation is block-wise scaling for weights and activations. Each 128x128 block of a weight matrix has its own scaling factor; each 128-element segment of an activation tensor has its own scale. The block size is small enough to capture local outliers, large enough to keep the scale-factor overhead modest.

The technical implementation: scale factors are stored alongside the tensor data, computed on the fly during forward and backward passes, and applied during the matmul. The kernel implementation is custom — DeepSeek wrote optimized CUDA for block-scaled FP8 matmul because Transformer Engine's per-tensor scaling didn't give enough precision for their model size.

Mixed-precision accumulation

DeepSeek's recipe accumulates the FP8 matmul result in FP32, then casts back to BF16 for the inter-layer activation. The accumulator promotion is essential; without it the recipe destabilizes within thousands of steps.

Loss-scaling-like compensation

For backward-pass FP8 gradients, DeepSeek's recipe includes an adaptive scale-factor adjustment that's structurally similar to FP16's loss scaling but operates per-tensor rather than per-loss. The implementation detail: at each step, if a tensor's observed max approaches the FP8 max, the scale factor is increased; if the observed max is well below the FP8 max, the scale is decreased.

Layer-specific exclusions

The recipe explicitly excludes the embedding, the LM head, layer norms, and softmax from FP8. These layers have outlier distributions or non-matmul-shaped compute that FP8 doesn't handle well.

What it cost to develop

DeepSeek-V3's training run is documented at around 2.78M H800-hours, which is unusually cheap for a frontier-scale model. A meaningful fraction of the cost discipline came from the FP8 recipe — roughly 2x throughput vs an equivalent BF16 run. Without FP8, the same training would have cost 5-6M H800-hours.

What other labs have learned from it

The block-wise scaling pattern has been adopted by several follow-up open recipes (Qwen 2.5 partial recipes, some Tülu work). The Transformer Engine team has incorporated block-scaling support into recent releases. The recipe is on its way to being the de facto frontier FP8 standard.

Llama 3 used BF16, not FP8 — why

A notable contrast: Llama 3 (Meta) used BF16 for its entire training run, not FP8. The published rationale: BF16 was the safer choice given Meta's engineering risk tolerance, and the throughput gain from FP8 was not deemed worth the stability risk for their pipeline. The choice has been retrospectively criticized as conservative — Llama 3.3 (also BF16) could likely have shipped at lower cost with FP8 — but it reflects the legitimate uncertainty about FP8 stability at frontier scale that existed before DeepSeek-V3 published.

The takeaway: in 2024-2025 there were two reasonable engineering positions on FP8 for frontier training. After DeepSeek-V3, the position favoring FP8 is stronger; frontier labs that haven't moved to FP8 are increasingly the outliers.

MS-AMP and torchao FP8: the other implementations

Transformer Engine is the dominant FP8 implementation but not the only one. The alternatives matter for teams using different stacks.

MS-AMP (Microsoft)

Microsoft's mixed-precision training library, with FP8 support added in 2023 ([arXiv:2310.18313]). Supports the same forward-E4M3/backward-E5M2 split as Transformer Engine; differs in scale computation (per-tensor with explicit calibration steps rather than Transformer Engine's continuous adaptation). Integrated with DeepSpeed via the FP8-LM pipeline.

Strengths: tight DeepSpeed integration, well-tested at multiple scales. Weaknesses: NVIDIA-only, less continuous adaptation than Transformer Engine.

torchao FP8

PyTorch's native FP8 support, in active development through 2025-2026. Implements per-tensor, per-row, and per-channel scaling. Less mature than Transformer Engine but increasingly the default for new PyTorch projects because it avoids the Transformer Engine dependency.

The 2026 status: torchao FP8 is production-viable for SFT and DPO at 70B scale. For frontier pretraining, Transformer Engine still has the throughput edge and the longer track record.

FP8 in JAX

JAX's Pallas kernels and the TPU-side FP8 support give a different implementation path for FP8 training. The TPU FP8 implementation is sufficiently different from NVIDIA's that recipes don't port directly; the formats are similar but the scaling logic and operator support diverge.

Implementation comparison

Implementation	Vendor	Scaling	Accumulator	Frameworks
Transformer Engine	NVIDIA	Per-tensor, history-based	FP32	PyTorch, JAX, Megatron-LM
MS-AMP	Microsoft	Per-tensor, explicit	FP32	DeepSpeed, PyTorch
torchao FP8	PyTorch	Per-tensor, per-row, per-channel	FP32	PyTorch native
DeepSeek custom	DeepSeek	Block-wise (128x128)	FP32	Custom
JAX/TPU FP8	Google	Per-tensor	FP32	JAX, Flax

The pattern across implementations: scaling strategy is the main differentiator; the accumulator is universally FP32; the format choice (E4M3 forward, E5M2 backward) is consistent.

FP8 with FSDP, TP, and pipeline parallelism

Distributed parallelism interacts with FP8 in subtle ways. Production frontier training combines them, so the interaction matters.

FP8 + FSDP2

PyTorch FSDP2 (the rewrite that landed in PyTorch 2.6) handles FP8 weights cleanly: shards are stored in FP8, gathered on-demand into BF16 for matmul (when using Transformer Engine), and rescattered. The communication volume of FSDP all-gathers drops by 2x relative to BF16 because the shards are half the size.

The gotcha: gradient all-reduces in FP8 are technically possible but introduce additional precision loss. Most production stacks all-reduce in BF16 or FP32 even when the matmul itself runs in FP8.

FP8 + tensor parallelism (TP)

Megatron-LM's TP partitions weight matrices across GPUs. Each rank holds a slice of the weights, performs matmul on its slice, and exchanges results via all-reduce or all-gather. FP8 weights work with TP; the exchanges happen in BF16 (the matmul output precision) by default.

The gotcha: per-tensor scale factors must be synchronized across TP ranks for each weight matrix. The standard approach is to compute the global max across all ranks and use it as the scale; computing per-rank scales independently leads to inconsistent quantization.

FP8 + pipeline parallelism (PP)

PP partitions layers across GPUs. Inter-stage activations cross GPU boundaries. The activations can be transmitted in FP8 (saving bandwidth) or BF16 (saving the scale-factor overhead). Most production setups transmit in BF16 because the activation precision matters for stability.

FP8 + 3D parallelism

At frontier scale (DeepSeek-V3, Llama 3 405B class), 3D parallelism (DP + TP + PP) is the norm. FP8 composes with all three, but the engineering complexity is substantial. The interaction surface area — scale-factor synchronization across TP, FP8 weights in FSDP shards, activation precision across PP boundaries — is where most frontier-scale FP8 bugs live.

Communication precision

Within a training step, multiple all-reduces happen: gradient all-reduce, parameter all-gather (FSDP), activation all-reduce (TP). Each can run in different precision. The 2026 default pattern:

Communication type	Precision
Gradient all-reduce	BF16 (saves bandwidth, minor stability cost) or FP32 (safer)
Parameter all-gather (FSDP)	FP8 if weights are FP8, else BF16
Activation all-reduce (TP)	BF16
Loss reduction	FP32
KV cache transfer (PP)	BF16

The art is choosing the highest precision needed for stability without leaving bandwidth on the table.

Per-model-size feasibility: where FP8 pays

The case for FP8 strengthens with model size; for small models it can be net-negative. A per-scale breakdown.

1B-class models

FP8 typically does not pay. Throughput gains are minor (small matmuls don't amortize the scale-factor overhead). Stability risks are real. BF16 is the right default.

7B-class models

FP8 is borderline. Throughput gains of 1.3-1.7x are achievable; engineering effort is substantial. Most teams stick with BF16 unless they're training many 7B models and the cumulative savings matter.

13B-class models

FP8 starts to pay clearly. Throughput gains of 1.5-1.8x. Most published recipes still use BF16, but FP8 is increasingly viable.

70B-class models

FP8 is the right default if the team has the engineering capacity. Throughput gains of 1.8-2.0x. Stability is well-understood at this scale.

100B-400B-class models

FP8 is essentially mandatory for cost efficiency. Throughput gains of 2.0-2.2x. Frontier labs run FP8 here as a matter of course.

671B+ (MoE total)

FP8 essential. DeepSeek-V3 (671B total, 37B active) demonstrates feasibility. The block-wise scaling becomes more important at this scale because activation distributions get more varied.

The cost-benefit curve

Model size	FP8 throughput gain	Stability risk	Engineering cost	Recommendation
1B	1.1-1.3x	Low	Same as 7B	BF16
7B	1.3-1.7x	Low-medium	Moderate	BF16 unless cost-driven
13B	1.5-1.8x	Medium	Moderate	FP8 if team has experience
30B	1.7-2.0x	Medium	Moderate	FP8 default
70B	1.8-2.0x	Medium	High	FP8 with care
200B+	2.0-2.2x	Medium-high	High	FP8 mandatory for cost
Frontier MoE	2.0-2.2x	High	Very high	FP8 with block scaling

The pattern: stability risk grows slowly with model size, throughput gain grows rapidly. The crossover where FP8 becomes worth it is around 13-30B for most teams; for cost-driven teams the crossover comes earlier.

MoE and FP8: the special case

Mixture-of-experts models add wrinkles to FP8 training that the dense-model recipes don't address.

Expert imbalance and scale calibration

In an MoE, different experts receive different amounts of traffic. Underused experts have less-calibrated FP8 scale factors than heavily-used experts. The standard fix is to compute scale factors across all expert weights jointly (a single global scale per layer) rather than per-expert, even though that wastes some precision on the heavily-used experts.

DeepSeek-V3's recipe uses per-expert scaling with explicit balancing of expert traffic during training (auxiliary loss for load balancing). The combination — block-wise per-expert FP8 plus traffic balancing — is what enables FP8 to work at the 671B MoE scale.

Routing precision

The router (the gating network that decides which expert handles each token) runs in BF16, not FP8. The routing decision is sensitive to numerical noise; quantizing it to FP8 introduces too much variance. The router is a small part of total compute, so the precision cost is manageable.

All-to-all and FP8

MoE training requires all-to-all communication (sending tokens to the experts that handle them). The all-to-all can run in FP8 (saving bandwidth) or BF16. Most production setups use FP8 for the forward all-to-all (tokens to experts) and BF16 for the backward all-to-all (gradients back). The asymmetry reflects the same value-distribution intuition as the E4M3/E5M2 split.

Cross-references

For the inference side of MoE serving, see mixture-of-experts serving. For the dense-model parallelism patterns that MoE training inherits, see distributed LLM training.

Fine-tuning in FP8 vs BF16

Fine-tuning is a different regime from pretraining and the FP8 decision is different.

The data is the bottleneck, not the compute

Most fine-tuning runs are bounded by data quality and quantity, not compute. The throughput gain from FP8 (1.5-2x) saves engineering time but doesn't change the achievable model quality.

Stability is easier in fine-tuning

Fine-tuning starts from pretrained weights with stable distributions. FP8 scale factors converge quickly to good values. The "destabilizes at step 50K" failure mode is rarer in fine-tuning because typical fine-tuning runs are much shorter than 50K steps.

LoRA + FP8

LoRA adapters can be in FP8 even when the base model is in BF16 or FP16. The adapter matmuls account for a small fraction of total compute; FP8 there saves modest amounts. Most production LoRA stacks keep adapters in BF16 for simplicity.

For full-parameter fine-tuning, FP8 is increasingly the default at 30B+ scale and starts becoming useful at 13B. For LoRA fine-tuning, FP8 rarely pays back the engineering complexity.

QAT for inference targets

A related but distinct topic: Quantization-Aware Training (QAT) trains a model in higher precision but with simulated INT8 or INT4 quantization noise injected, so the resulting model degrades less when actually quantized for inference. QAT is independent of FP8 training; you can run QAT on a BF16 trainer or an FP8 trainer. The use case is preparing a model for INT4 weight-only inference deployment — see quantization tradeoffs.

Learning-rate schedules and precision interactions

The interaction between learning rate schedules and FP8 precision is more subtle than it appears. Schedules that work in BF16 may need adjustment in FP8.

Warmup and FP8

The first few thousand steps of training have high gradient variance. FP8 quantization noise compounds with this variance, increasing the risk of instability during warmup. The standard fix: do warmup in BF16, switch to FP8 after warmup is complete (typically after 1-3K steps).

Some recipes do warmup in BF16 and then switch to FP8 with a brief BF16-to-FP8 transition period where the scale factors are calibrated against the current weight distribution. The transition takes 100-500 steps.

Peak learning rate

FP8 runs typically use the same peak learning rate as BF16 — the throughput speedup doesn't change the optimal step size. Some teams have reported needing a slightly lower peak LR in FP8 (10-30% lower) to maintain stability; the evidence is mixed and likely model-specific.

Cosine decay and FP8

Cosine learning-rate schedules work fine with FP8. The decay phase, where gradients become smaller, is the time when FP8 underflow risk increases. Late-training stability checks are particularly important.

LR rewind and resume

Restarting training from a checkpoint with a different LR schedule (LR rewind) interacts with FP8 the same way it interacts with BF16. The scale factors will re-adapt to the new gradient magnitudes; budget 200-500 steps for the scale factors to settle.

FP8 and gradient accumulation

Gradient accumulation — running multiple micro-batches and accumulating gradients before the optimizer step — interacts with FP8 in specific ways.

Where the accumulation happens

The accumulation buffer is typically FP32. Each micro-batch produces FP8 gradients (in E5M2 typically); those gradients are upcast to FP32 and added to the accumulation buffer. After the configured number of micro-batches, the accumulated FP32 gradient is used for the optimizer step.

The upcast-and-accumulate pattern is what makes FP8 gradient accumulation work without precision loss. Skipping the upcast (accumulating in FP8 directly) destabilizes within hundreds of steps.

Memory implications

The FP32 gradient accumulation buffer adds memory cost. For a 70B model with FSDP2 and 16-way gradient accumulation, the buffer is shared across all micro-batches in a step; total cost is around 280GB across the cluster (sharded). For most production setups this is acceptable.

Loss scaling for gradient accumulation

When using gradient accumulation, the loss is typically divided by the number of accumulation steps to compute the average gradient. This division can interact with FP8 scaling — the loss-divided-by-N is a smaller number, which may underflow at the very last accumulation step. The standard mitigation is to do the loss division in FP32 after the accumulation, not before.

Communication precision in detail: BF16 vs FP32 all-reduce

Distributed training does several types of communication; each has a precision choice with stability implications.

Gradient all-reduce: BF16 vs FP32

Gradient all-reduce in FSDP and DDP averages gradients across data-parallel ranks. The bandwidth cost is proportional to the precision. BF16 all-reduce uses half the bandwidth of FP32 all-reduce.

The stability trade-off: BF16 all-reduce has lower precision per gradient element. For most workloads, this is fine — the gradient noise from the all-reduce is smaller than the gradient noise from the underlying mini-batch sampling. For very large worlds (1024+ ranks), the cumulative precision loss matters and FP32 all-reduce is safer.

A useful heuristic: BF16 all-reduce up to 512 ranks, FP32 all-reduce at 1024+ ranks. Frontier-scale (10K+ ranks) almost always uses FP32.

NCCL precision modes

NCCL (the NVIDIA collective communications library) supports all common precisions. The choice is made by the framework; most provide a flag for BF16 vs FP32 communication. For deeper coverage of NCCL tuning, see NCCL guide.

FP8 communication?

FP8 all-reduce is technically supported by some implementations but not widely used. The precision loss compounds across the reduction tree and the gains are modest (half the bandwidth of BF16). Most production stacks reserve FP8 for the matmul path and use BF16 or FP32 for communication.

Cross-references

For the underlying networking layer (InfiniBand vs RoCE, congestion control, topology), see AI training networking and NVLink and rack-scale topology.

Numerical-failure taxonomy: diagnosing FP8 runs

When an FP8 run misbehaves, the failure mode usually falls into one of a small set of recognizable patterns. Knowing the taxonomy speeds debugging dramatically.

Pattern 1: Immediate NaN

Loss is NaN within the first few steps. Cause: a layer in the model has outlier values that overflow FP8 before the scale factor has had a chance to adapt. Diagnosis: log per-layer max values during the first 100 steps. Fix: warm up the scale factors with a few hundred steps in BF16 before switching to FP8, or use a more aggressive initial scale factor.

Pattern 2: Sudden divergence at step N

Loss is healthy for 10K-50K steps, then suddenly diverges. Cause: weight distribution has drifted out of the FP8 scale's range. Diagnosis: log per-layer scale factors over training; the layer whose scale spiked just before divergence is the culprit. Fix: shorter scale history window, per-block scaling for that layer, or exclusion of that specific layer from FP8.

Pattern 3: Slow quality degradation

Loss curves look reasonable but eval scores are systematically lower than the BF16 baseline. Cause: per-tensor scale factors are not granular enough to capture local outlier patterns; the resulting quantization noise accumulates. Diagnosis: ablate FP8 layer-by-layer to find the offender. Fix: per-row or per-block scaling for the affected layer.

Pattern 4: Throughput regression

FP8 is supposed to be faster than BF16 but the wall-clock per step is the same or slower. Cause: layers too small to benefit from FP8 (the scale-factor overhead dominates the matmul throughput gain), or kernel implementation that's not actually FP8-fused. Diagnosis: profile per-layer kernel times. Fix: keep small layers in BF16, verify that the FP8 kernels are actually being dispatched.

Pattern 5: Gradient norm explosion

Gradient norm grows monotonically across training, gradient clipping fires constantly. Cause: backward-pass FP8 scale factor is too aggressive, gradients are quantizing into the high end of the E5M2 range. Diagnosis: log backward-pass scale factors and per-layer gradient norms. Fix: more conservative scale factor for backward pass, or relax to BF16 for the backward pass entirely.

Pattern 6: Loss oscillation

Loss fluctuates wildly without trend. Cause: scale factors oscillating between values, causing alternating overflow and underflow. Diagnosis: scale-factor logs show high variance. Fix: longer history window for scale-factor computation, or explicit scale-factor smoothing.

Pattern 7: Checkpoint-restore divergence

Training resumes from a checkpoint and immediately diverges, even though the saved state was healthy. Cause: scale factors not saved with the checkpoint, or saved scale factors not loaded correctly. Diagnosis: compare scale factors before checkpoint and after restore. Fix: ensure scale factors are part of the checkpoint state.

A diagnostic dashboard

A production FP8 run should have a dashboard with:

Loss (linear and log scale)
Per-layer scale factor for E4M3 weights, E4M3 activations, E5M2 gradients
Per-layer max absolute value of weights, activations, gradients
Underflow ratio (fraction of values quantizing to zero) per layer
Gradient norm (global)
Throughput (samples/sec)

The patterns above are diagnosable in 10 minutes with such a dashboard and an afternoon without it.

Checkpoint precision: what to save and reload

A surprisingly common source of bugs is mixed-precision checkpointing. The training-time precisions don't all need to persist; the question is which ones must.

What must be saved in FP32

Master weights. Every reliable recipe stores these in FP32.
Optimizer state (Adam m and v). Must be FP32 for numerical accuracy.
Loss scaler state (for FP16 runs) or scale factors (for FP8 runs).

What can be saved at lower precision

Step counter, learning rate schedule state, RNG state. These are not precision-sensitive.
Activation checkpoint state (if any). Not typically persisted across job restarts.

What about the deployed model?

The model checkpoint for serving is typically saved in BF16 (a downcast from the FP32 master). For some deployments, FP8 weights are saved directly to avoid the runtime quantization step. The choice depends on the serving stack; FP8 deployment is well-supported by vLLM, SGLang, and TensorRT-LLM in 2026.

Checkpoint size implications

A 70B model in FP32 master + FP32 optimizer state + FP32 momenta is around 1.4TB. Reducing optimizer state to FP16 (an experimental optimization) drops this to about 900GB. Most production stacks accept the 1.4TB checkpoint size in exchange for the FP32 numerical stability.

For background on checkpoint infrastructure, see checkpoint storage and recovery.

torch.compile and FP8: the interaction

torch.compile changes the kernel-fusion picture and interacts with FP8 in specific ways worth understanding.

What torch.compile does for FP8

When applied to a model using Transformer Engine FP8, torch.compile can fuse the surrounding operations (layer norm, residual additions, activations) with the FP8 matmul, reducing memory bandwidth requirements. The throughput gain is workload-dependent but typically 5-15% on top of the FP8 speedup.

What torch.compile breaks for FP8

The dynamic graph capture is sensitive to control flow. Transformer Engine's scale-factor update logic includes some control flow (skip the update if the tensor is all zeros, for example). torch.compile may not capture this correctly, leading to stale scale factors in compiled paths.

The pragmatic solution: compile the parts of the model that don't include scale-factor updates (the matmul and surrounding fusion targets) and leave the scale-factor management uncompiled. PyTorch 2.6+ has improved support for this pattern.

Compile and FSDP2

torch.compile interacts with FSDP2 in subtle ways. The shard-gather-matmul-rescatter pattern can be compiled end-to-end in some configurations, breaking the FSDP2 abstraction. The 2026 best practice: compile the local matmul portion but leave the FSDP collective management uncompiled.

For deeper coverage of torch.compile and CUDA graphs, see CUDA graphs and torch.compile.

Worked example: budget for a 70B FP8 vs BF16 run

Concrete numbers for the FP8 vs BF16 trade at frontier scale.

Setup

A 70B dense model, 8K context, 1T training tokens. Training on a 256xH100 cluster.

BF16 baseline

Per-step time: 8 seconds. Step count for 1T tokens: roughly 130K steps. Total wall clock: 290 hours = 12 days. Total H100-hours: 290 * 256 = 74K H100-hours. Cost at $2.50/hr: $185K.

FP8 (Transformer Engine HYBRID, no block scaling)

Per-step time: 4.5 seconds (1.78x speedup). Step count: same 130K. Wall clock: 163 hours = 6.8 days. H100-hours: 42K. Cost: $105K.

FP8 (DeepSeek-style block scaling)

Per-step time: 4.0 seconds (2.0x speedup over BF16). Step count: same. Wall clock: 144 hours = 6 days. H100-hours: 37K. Cost: $92K.

Savings summary

Precision	Wall clock	H100-hours	Cost (compute)	Quality cost
BF16	12 days	74K	$185K	Baseline
FP8 standard	6.8 days	42K	$105K	Within 0.1 points on MMLU
FP8 block-scaled	6 days	37K	$92K	Within 0.05 points

The cost savings are substantial — $90K+ on a single run — and the quality cost is small. The engineering investment to set up FP8 properly amortizes after one or two runs.

What the numbers don't include

Failed run cost. A run that diverges at step 50K wastes 19 days of compute. FP8's higher stability risk means more failed runs in the learning phase.
Engineering time. Setting up FP8 correctly takes weeks of engineering effort the first time.
Eval and ablation runs. Production training typically does many ablation runs at smaller scale to validate the recipe. Each ablation pays the FP8 setup cost.

The total cost of moving a team's training stack from BF16 to FP8 is typically 3-6 months of one engineer's time plus several weeks of cluster time on failed runs. Worth it for teams running many large training runs per year; less obviously worth it for teams running one or two runs.

The bottom line

The named problem is the numerical-range cliff: FP32 is too expensive to leave compute on the table, but FP16 and FP8 silently underflow gradients and crash training. The solution is mixed precision — pick the right format for each role, with FP8/BF16 in the hot matmul path, FP32 master weights and optimizer state outside it, and per-tensor scaling to make every FP8 tensor use its full range. The single biggest lever is keeping the FP32 master weights and Adam moments — every reliable FP8 recipe in production keeps them, every recipe that drops them eventually diverges.

What to do if you take only this away:

Default to BF16 mixed precision. It needs no loss scaling, no calibration, and works on every modern accelerator.
Move to FP8 only on Hopper or newer, for 1B+ models, and always with per-tensor scaling (Transformer Engine, MS-AMP, or framework equivalent).
Exclude the LM head, embedding, layernorm, and softmax from FP8 — these are the layers that produce the outliers that break calibration.
Audit your run with gradient-norm and underflow-ratio dashboards before assuming convergence; FP8 failures show up at step 50k, not step 500.
Treat FP4 as inference-only in 2026; FP4 training is research, not production.

Next, read NVIDIA datacenter GPUs for the tensor-core throughput each format unlocks, and distributed LLM training for how mixed precision composes with FSDP, TP, and gradient accumulation.

FAQ

Q: Should I always use FP8?

For frontier-scale training: yes, the speedup compounds. For research/small runs: BF16 is simpler and the speedup may not be worth the operational overhead.

Q: BF16 vs FP16, what's the difference?

BF16 has FP32's dynamic range with half the precision bits. FP16 has narrower dynamic range and needs loss scaling. BF16 is strictly easier to train with.

Q: Why is FP4 still risky?

Less data on quality at scale. Frontier labs are still learning where it works and where it doesn't. By 2027 it should be more standard.

Q: Can I mix BF16 and FP8 layers?

Yes, and you should. Most production FP8 training keeps numerically-sensitive layers (LayerNorm, softmax) in BF16 while running MLP matmuls in FP8.

Q: How do I handle FP8 in inference vs training?

Different concerns. Inference uses FP8 weights + FP8 KV typically; calibration is done at deployment time, not during inference. Training has both forward and backward in FP8 with delayed scaling.

Q: What about INT8 training?

Doesn't work as well as FP8 for forward/backward. Has its niche in inference quantization-aware training. Rarely used for full training.

Q: TF32 — what is that?

A 19-bit format (stored in 32-bit) on Ampere+ tensor cores. Faster than FP32, similar dynamic range. Default for FP32 matmuls on those GPUs. Not really mixed precision; more "FP32 made faster."

Q: Does FP8 work with FSDP/DeepSpeed/Megatron?

Yes. NVIDIA's libraries integrate Transformer Engine; PyTorch FSDP gained native FP8 support in late 2024.

Q: How does FP8 interact with quantization-aware fine-tuning?

QAT for inference quantization happens after training. Mixed-precision training is during training. They're separable. Some advanced setups train with FP8 forward and then deploy with INT4 weights.

Q: When does it make sense NOT to use mixed precision?

For very small models or research toy problems where simplicity beats throughput. Otherwise, use BF16 minimum.

Q: How do I debug training divergence after enabling FP8?

Step 1: confirm calibration is enabled. Without per-channel scaling, FP8 quality degrades sharply.

Step 2: check for NaN/Inf in gradients. Frequent NaNs = numerical issues. Lower learning rate, increase grad clipping.

Step 3: identify which layer is unstable. Check per-layer activation statistics. Switch problematic layers to BF16.

Step 4: extend amax_history_len to smooth out outliers.

Step 5: as a last resort, fall back to BF16 for the entire run. Investigate FP8 issues offline.

Q: What about FP8 for training reasoning models (o1, R1)?

Works fine for the underlying transformer training. The reasoning-specific aspects (long thinking sequences, RL post-training) are orthogonal to numeric precision.

DeepSeek-R1 was trained with FP8 base + FP32 RL adjustments. Standard pattern in 2026.

Q: Are there models that can't use FP8?

Some attention configurations are numerically sensitive (high-temperature softmax, very-long-context retrieval). For these, FP8 may need BF16 fallback for attention layers.

If you see >0.5 quality drop after enabling FP8, the architecture may not be FP8-friendly. Investigate.

Q: How does mixed precision interact with quantization-aware training (QAT)?

QAT is for inference quantization. It trains the model to be robust to deployment-time quantization (e.g., INT4 weights for edge deployment).

Forward pass during QAT can use FP8 (training optimization) or BF16 (simpler). Backward and optimizer state stay FP32.

After QAT, deploy in INT4. The trained model has activations that quantize cleanly.

Q: What's the right approach for fine-tuning?

Standard mixed precision: BF16 weights and gradients, FP32 master and optimizer.

For LoRA fine-tuning, the LoRA adapters can be even lower precision (INT8) without quality issues. The base model stays in BF16/FP16.

Q: Should I quantize the optimizer state?

Some experimental setups use 8-bit Adam (bnb's BNB). Saves ~50% optimizer memory.

Quality impact: usually small (~0.1 points). For memory-constrained training, worth the trade.

For frontier-scale training: stay with FP32 optimizer state. Memory savings rarely justify the small quality cost.

Q: What's coming after FP8 / FP4?

Microsoft's MX formats (MXFP6, MXFP4) with shared exponents. Can offer better quality at the same bit width.

OCP (Open Compute Project) is standardizing these formats. Expect industry-wide support over 2026-2027.

Beyond: ternary, binary networks. Useful for specific deployments but not for frontier training.

Q: How do I test mixed precision setups?

Three-step validation:

Eval on a benchmark (MMLU, HellaSwag) at BF16 baseline. Note results.
Re-train (or re-eval) with FP8. Check that benchmark scores are within 1 point.
Long-run stability test (10k+ steps). Watch for divergence or quality drift.

If passes all three: ship.

Q: Should I use per-tensor or per-row scaling for FP8?

Per-tensor is the default and works well for most workloads. Per-row scaling (one scale per row of the weight matrix) helps when there are outlier columns in the weight matrix — common in attention projections after long training. DeepSeek-V3 uses block-wise scaling (per 128x128 block) which is even finer than per-row. The right answer scales with how outlier-prone the model's distributions are: more outliers, finer scaling.

Q: Why does FP8 destabilize at step 50K instead of step 500?

The slow-onset failure mode is usually drift in the weight distribution. Scale factors that were correct early in training stop being correct after enough updates accumulate. The fix is more aggressive scale-factor adaptation (shorter history window) or per-block scaling that captures local distribution changes.

Q: Does FP8 affect the optimizer at all?

No, when implemented correctly. The optimizer (Adam, AdamW, Lion, etc.) operates on FP32 master weights and FP32 moments. FP8 affects only the matmul precision in forward and backward; the optimizer step is unchanged.

Q: Can I use FP8 for the entire model including embeddings and LM head?

Not safely. Embeddings and the LM head are the layers with the most outlier-prone distributions. Production recipes (DeepSeek-V3, MS-AMP, Transformer Engine defaults) all keep these layers in BF16. The throughput cost is minor (these layers are a small fraction of total compute).

Q: How does FP4 compare to FP8 in terms of stability risk?

FP4 has roughly 2x the throughput of FP8 and 4-10x the stability risk. The narrow dynamic range (about 8 distinct values per block) means small distribution shifts can blow up the scaling. Production FP4 training (as of mid-2026) is experimental; FP4 inference is more mature. The Blackwell hardware support is excellent; the recipe maturity is the limiting factor.

Q: What's MXFP4 vs NVFP4?

Both are FP4 formats with block-wise scaling. NVFP4 is NVIDIA-proprietary; MXFP4 is the Open Compute Project standard. Differences include block size (NVFP4 uses 16, MXFP4 uses 32 typically) and shared scale type (NVFP4 uses E4M3, MXFP4 uses E8M0). Hardware support varies by vendor. For cross-vendor portability MXFP4 is preferable; for NVIDIA-only deployment NVFP4 has slightly better hardware support today.

Q: Does FP8 training work on AMD MI300X?

Partially. AMD's matrix cores support FP8 with similar throughput characteristics to NVIDIA. The software stack (ROCm + flash attention + FP8 kernels) is less mature than CUDA + Transformer Engine. Production FP8 training on MI300X is feasible for SFT and DPO but not yet for frontier pretraining. The gap is closing; 2027 is a reasonable target for parity.

Q: Does FP8 affect determinism?

Slightly. FP8 quantization introduces small numerical variations that compound across many steps. Strict bitwise determinism is harder to achieve in FP8 than BF16. For most production training the precision tradeoff for throughput is worth the small loss in determinism; for debugging runs that need reproducibility, BF16 is safer.

Q: Can I train a model in FP8 and serve it in FP16/FP8 with no quality loss?

Mostly yes. A model trained in FP8 typically serves well in FP8 (the distributions are already in the FP8 range) and in BF16 (which is a precision upgrade). Serving in FP16 requires careful range checking because FP16 has narrower dynamic range than BF16; outlier activations can overflow. The conventional pattern: train in FP8, serve in FP8 or BF16, avoid FP16 for serving FP8-trained models.

Q: How does FP8 interact with gradient clipping?

Carefully. Gradient clipping computes a global norm and rescales gradients. The norm computation must happen in BF16 or FP32 (not FP8) because the norm can be a large number that doesn't fit in FP8's range. Most implementations cast to FP32 for the norm computation, then apply the clipping to the FP8 gradients before backward continues.

Q: What's "BAcc" and why does it matter?

BAcc is the block accumulator — the precision in which partial products of an FP8 matmul are summed before the result is produced. Production implementations always use FP32 for BAcc. Running matmuls with pure-FP8 accumulation (no BAcc promotion) destabilizes training within hours. The implementation detail is hidden from the user in Transformer Engine but matters for understanding why FP8 is more than just "use FP8 everywhere."

Q: Does FP8 training affect activation checkpointing?

Slightly. Activation checkpointing stores activations during forward and recomputes during backward to save memory. The stored activations can be in BF16 (the standard) or FP8 (saves more memory at the cost of recomputation precision). Most production stacks store in BF16 even when matmuls run in FP8 because the precision matters for the backward recomputation.

Q: How does FP8 compose with selective activation checkpointing?

Selective activation checkpointing (only some layers are recomputed, others are kept in memory) interacts with FP8 the same way standard activation checkpointing does. The activations stored in memory can be BF16 or FP8 depending on memory pressure; the activations recomputed during backward should match the original forward precision.

Q: Are there published case studies of FP8 training going wrong?

Yes, in research follow-ups but few in production. The published DeepSeek-V3 paper is unusually forthcoming about what didn't work in their FP8 development; several follow-up papers (block-wise scaling, FP4 pretraining experiments) document failure modes. The pattern across published failures: scaling factor not adapting fast enough, layer exclusion list too narrow, gradient distribution drift after many epochs.

Q: When will FP4 training be production-ready?

For frontier-scale dense pretraining: probably 2027-2028. For SFT and fine-tuning on stable weights: late 2026 is plausible. The bottleneck is recipe maturity, not hardware — Blackwell's FP4 support is excellent. The conservative bet: stay on FP8 for production training through 2026, evaluate FP4 in 2027 once the first published frontier-scale FP4 training run lands.

Long-form analysis: when each format wins

Detailed analysis of when each precision format is the right choice.

FP32: when (almost never)

FP32 was the standard before mixed precision. In 2026, few use cases:

Numerical research where reproducibility across hardware matters.
Algorithms specifically requiring high precision (some optimization research).
Debugging mixed-precision issues.

For production training: never. The compute and memory cost is prohibitive.

TF32: when

NVIDIA's TF32 is essentially "FP32 with FP16 mantissa precision." Auto-enabled on Ampere+ for FP32 ops.

When to use: never explicitly. PyTorch and others use TF32 transparently when FP32 is requested. You don't need to think about it.

FP16: when (rarely now)

FP16 was the original mixed-precision format. Has narrow dynamic range; needs loss scaling.

When to use:

Hardware without BF16 (older GPUs).
Specific algorithms designed for FP16.

When not to use: anywhere BF16 is available. BF16 is strictly easier.

BF16: when (most cases)

The safe modern default for training.

When to use:

Most pre-training and fine-tuning.
When FP8 quality concerns matter.
For research where simplicity beats throughput.

When not to use:

Memory-constrained training (FP8 saves more memory).
Frontier-scale where FP8's throughput gain matters.

FP8 e4m3 / e5m2: when (frontier production)

Modern default for frontier training on Hopper+.

When to use:

Pre-training large models where compute scales.
Fine-tuning where speed matters.
Memory-constrained training.

When not to use:

Very small models (overhead dominates gain).
Workloads with known FP8 quality issues.
Hardware without FP8 (Ampere or older).

FP4 (Blackwell): when (emerging)

Cutting-edge in 2026. Some research training, some inference.

When to use:

Frontier-scale on Blackwell where you can validate quality.
Inference where FP4 quality is acceptable.

When not to use:

Production training without extensive validation.
Quality-critical workloads.
Anywhere outside Blackwell.

INT8: when (legacy)

Pre-FP8 alternative to BF16 for training.

When to use:

Older hardware that doesn't support FP8.
Inference quantization with QAT.

When not to use:

Modern Hopper+ training. FP8 is better.

INT4: when (memory-constrained)

For inference deployments where memory is the binding constraint.

When to use:

Inference on memory-limited GPUs (consumer 4090, smaller datacenter SKUs).
Edge deployment.
Cost-optimized serving.

When not to use:

Quality-critical workloads.
Training (causes too much numeric drift).
Long-context retrieval (INT4 KV breaks).

Picking by workload

Frontier pre-training: FP8 mixed. Frontier fine-tuning: FP8 if speed matters, BF16 if simplicity. Production inference, normal: FP8 weights + FP8 KV. Production inference, memory-constrained: AWQ INT4 weights. Production inference, long-context retrieval: BF16 weights + FP8 KV. Edge inference: NF4 / Q4_K_M. Research / experimentation: BF16.

This matrix covers most cases. Specific workloads may justify variants.

Specific quirks and pitfalls

Detailed pitfalls beyond the basic mistakes.

Layer norm in low precision

LayerNorm computes mean and variance across hidden dim. In FP8/FP4, accumulated values can overflow.

Solution: keep LayerNorm in BF16 even when other ops are in FP8.

Softmax in low precision

Attention softmax: exp(score) / sum(exp(score)). Can overflow for large scores.

Solution: subtract max before exp (numerical stability trick). FlashAttention does this.

Cross-entropy loss

Loss computed in FP32 even when forward is FP8. Numerical reasons.

PyTorch handles this automatically. Don't override unless you have a specific reason.

Gradient accumulation across batches

Accumulate in FP32. Quantize for optimizer step.

Without this: rounding errors accumulate over many micro-batches.

Optimizer-state drift in low precision

If Adam's m and v are stored in FP16/BF16, they slowly diverge from FP32 ground truth.

Solution: keep optimizer state in FP32. Only quantize the forward/backward pass.

Weight decay in low precision

Weight decay subtracts a fraction of weight from itself each step. In FP8, the subtraction may underflow if weights are near zero.

Solution: apply weight decay in FP32 master weights, then sync to FP8 weights.

Numerical comparison across precisions

If your code compares model outputs across precisions (e.g., for testing): expect differences. FP8 vs BF16 outputs match within ~0.001. Don't expect bit-identity.

Mixed precision across layers

When some layers use FP8 and others BF16, watch for tensor format conversions. Cast operations have small overhead but more importantly can break numeric expectations.

Use Transformer Engine to handle this transparently.

Initialization issues

Some initialization schemes (very small Xavier) underflow in FP8. The model starts with effectively zeroed weights.

Solution: scale initialization by 1.5-2× when training in FP8 e4m3.

Specific bugs in framework versions

PyTorch, Megatron, DeepSpeed all have had FP8-specific bugs. Watch for:

Incorrect quantization of weight gradients.
Lost precision in optimizer step.
Race conditions in delayed scaling.

Pin known-good framework version. Don't auto-update.

These pitfalls are rarely catastrophic individually but compound. Production teams know them.

A capacity planning case study

Real example of choosing precision for a training run.

Setup

Goal: pre-train a 70B-parameter model on 1.5T tokens. Hardware: 64× H100 (8 nodes × 8 GPUs). Budget: 14 days wall-clock.

BF16 calculation

Weights: 140 GB. Memory cost.
TP=8 + DP=8 fits.
Throughput: ~3,200 tokens/sec/GPU = ~205,000 aggregate.
Time: 1.5T / 205,000 / 86,400 = 85 days.

Doesn't meet 14-day budget. Need more GPUs or more speed.

FP8 calculation

Same memory profile (FP8 weights save some, but optimizer dominates).
Throughput: ~5,800 tokens/sec/GPU = ~370,000 aggregate.
Time: 1.5T / 370,000 / 86,400 = 47 days.

Closer but still doesn't fit 14 days. Need more GPUs.

FP8 + 3× more GPUs

192× H100.
Throughput: ~1.1M aggregate.
Time: ~16 days.

Fits with margin. Cost: 192 × $4/hr × 14 × 24 = $258k.

Decision

Run with 192 H100s + FP8. Saves vs more GPUs at BF16.

If FP8 quality validates: ship. If not: extend timeline or add more GPUs.

This kind of math drives real procurement decisions. Precision choice has real economic implications.

Practical Transformer Engine usage

Detailed examples for production TE usage.

Basic FP8 training setup

import torch
import torch.nn as nn
import transformer_engine.pytorch as te
from transformer_engine.common.recipe import DelayedScaling, Format

class TransformerBlock(nn.Module):
    def __init__(self, hidden_size, num_heads):
        super().__init__()
        self.attn = te.MultiHeadAttention(
            hidden_size, num_heads,
            params_dtype=torch.bfloat16
        )
        self.mlp = te.LayerNormMLP(
            hidden_size, hidden_size * 4,
            params_dtype=torch.bfloat16
        )

# Configure FP8 recipe
fp8_recipe = DelayedScaling(
    fp8_format=Format.HYBRID,  # e4m3 forward, e5m2 backward
    amax_history_len=16,
    amax_compute_algo="max",
)

# Forward pass with FP8
with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
    output = model(input)
    loss = compute_loss(output)

loss.backward()
optimizer.step()

Gradient accumulation pattern

optimizer.zero_grad()
for micro_step in range(grad_accum_steps):
    micro_batch = get_next_batch()
    
    with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
        loss = model(micro_batch) / grad_accum_steps
    
    loss.backward()  # accumulates gradients in FP32

# Clip gradients in FP32
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()  # FP32 master weights, then sync FP8 weights

Combined with FSDP

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.fully_sharded_data_parallel import ShardingStrategy

# Wrap with FSDP
model = FSDP(
    model,
    sharding_strategy=ShardingStrategy.FULL_SHARD,
    mixed_precision=MixedPrecision(
        param_dtype=torch.bfloat16,
        reduce_dtype=torch.float32,  # gradient reduction in FP32
    ),
)

# FP8 still applies inside FSDP boundary
with te.fp8_autocast(enabled=True):
    loss = model(input)

Combined with TP via Megatron

# Megatron config
config = {
    "tensor_model_parallel_size": 8,
    "fp8": "hybrid",
    "fp8_recipe": "delayed",
    "fp8_amax_history_len": 16,
}

# Megatron handles FP8 + TP integration internally
trainer = MegatronTrainer(model, config)

Layer-specific FP8

For layers where FP8 quality is insufficient, override:

from transformer_engine.pytorch.fp8 import fp8_autocast

# Most layers in FP8
with fp8_autocast(enabled=True):
    hidden = self.attention(x)
    hidden = self.mlp(hidden)

# Specific layer in BF16 (e.g., final layer norm)
with fp8_autocast(enabled=False):
    final = self.final_layer_norm(hidden)

This allows per-layer precision tuning when needed.

Quality validation in production

Beyond initial validation, ongoing quality monitoring during training.

During training

Track:

Loss per step (smooth decline expected).
Gradient norms per layer (stable).
Activation statistics (mean, std, max).
FP8 scale factors (stable across iterations).

Anomalies indicate FP8 issues.

Periodic eval

Every 1000-5000 steps:

Run a small eval suite (MMLU subset, custom benchmarks).
Compare to BF16 baseline at same step count.
If FP8 lags by > 2%, investigate.

A/B comparison

For frontier-scale runs, train a small (1B) model in BF16 alongside the FP8 70B run. Compare loss curves.

If FP8 loss curve diverges from BF16 reference: scaling issue.

Production deployment validation

Before deploying a trained model:

Eval on production-relevant benchmarks.
Compare to vendor-published numbers (Llama paper, Mistral paper).
Test on representative production traces.

If FP8 model underperforms: debug calibration, possibly retrain segments in BF16.

Long-tail quality metrics

Standard benchmarks (MMLU) don't catch all issues. Add:

Long-context retrieval (RULER).
Reasoning benchmarks.
Domain-specific evals.

FP8's quality cost varies by metric. Standard MMLU may be fine while RULER drops noticeably.

Comparing FP8 implementations across vendors

How NVIDIA, AMD, and others handle FP8.

NVIDIA Hopper / Blackwell

Native FP8 tensor cores. Two formats (e4m3, e5m2). Transformer Engine library handles calibration.

Most mature; production-ready.

AMD MI300/MI355

Native FP8 tensor cores since late 2024. Compatible numerics with NVIDIA.

ROCm software ecosystem still catching up. Performance competitive.

Intel Gaudi 3

Native FP8. Good performance for many workloads.

Smaller ecosystem; deployment limited.

Cerebras CS-3

FP8 support via wafer-scale architecture. Different numerics than mainstream.

Niche but capable for some workloads.

Standardization

OCP (Open Compute Project) is standardizing FP8 (and other low-precision) formats. Vendor compatibility improving.

By 2027-2028: cross-vendor FP8 portability should be mature.

For now: vendor-specific tooling. Cross-vendor isn't seamless.

Migration playbook: from BF16 to FP8

Step-by-step migration of a production training run.

Phase 1: prerequisites

Hardware: H100+ for FP8.
Software: CUDA 12.4+, Transformer Engine, PyTorch 2.4+.
Validation infrastructure: eval suite, comparison runs.

Phase 2: initial test

Train a small (1B) model in both BF16 and FP8 in parallel. Compare:

Loss curves at same step count.
Eval scores at end.

If they match within 1-2%: FP8 is healthy for your model.

Phase 3: scale up

Apply FP8 to fine-tunes of the small model. Validate quality preserves.

Phase 4: full migration

Switch your production training to FP8. Watch metrics carefully.

Phase 5: ongoing monitoring

After migration:

Periodic eval comparing to BF16 reference.
Watch for any quality drift.
Update FP8 recipe if quality changes.

Common migration issues

Quality regression > 1 point: calibration insufficient. Revisit.

Training unstable: gradient clipping may need to be tighter. Or specific layers need BF16 fallback.

Throughput gain < expected: profile. May be other bottleneck (data loader, network).

NaN in gradients: FP8 overflow. Check loss scaling, recipe, calibration data.

When to revert

If quality issues persist, revert to BF16. Don't ship a model trained with broken FP8.

The migration is reversible. Don't sunk-cost into a bad FP8 setup.

Q: What about training on AMD MI300/MI355 with FP8?

AMD's ROCm supports FP8 since late 2024. Most PyTorch + DeepSpeed training pipelines work on AMD.

Performance is competitive with H100 for many workloads. Software ecosystem (Megatron, NeMo) lags but Liger Kernel and similar fill the gap.

For new training projects in 2026, AMD is a viable alternative if cost matters and software constraints are acceptable.

Comparing major training framework FP8 implementations

Megatron-LM

NVIDIA's reference. Tightly integrated with Transformer Engine.

# Megatron-LM with FP8
megatron_args = {
    "fp8": "hybrid",  # e4m3 forward, e5m2 backward
    "fp8_recipe": "delayed",
    "fp8_amax_history_len": 16,
}

Pros: best peak performance. Most-tested at frontier scale. Cons: complex configuration. Megatron-specific knobs.

DeepSpeed

Microsoft's framework. FP8 support via Transformer Engine integration.

{
  "bf16": {"enabled": false},
  "fp8": {
    "enabled": true,
    "amax_compute_algo": "max",
    "amax_history_len": 16
  }
}

Pros: easier configuration. Good ZeRO + FP8 integration. Cons: slightly behind Megatron on raw FP8 performance.

PyTorch FSDP

Native PyTorch. FP8 via torchao or Transformer Engine wrapper.

from torchao.float8 import convert_to_float8_training
model = convert_to_float8_training(model)

Pros: PyTorch-native. Composes with FSDP-2 cleanly. Cons: less optimized than Megatron for frontier workloads.

NeMo

NVIDIA's high-level framework. FP8 enabled with one flag.

trainer:
  precision: bf16-mixed
  fp8:
    enabled: True
    fp8_format: hybrid

Pros: easy to start. Sane defaults. Cons: less flexible for unusual setups.

For most teams in 2026: NeMo for ease, DeepSpeed or Megatron for control, FSDP for PyTorch-native.

Production validation patterns

How frontier labs validate mixed-precision training.

Continuous quality benchmarks

Run a small eval suite every N training steps (e.g., every 1000):

MMLU subset (~500 questions).
HellaSwag subset.
Any custom evals.

Track loss + benchmark scores together. If they diverge (loss drops but benchmark drops too), something's wrong.

Loss curve comparison

Train a small (1B) model in both BF16 and FP8 in parallel. Compare loss curves at the same step count.

If they track within 1-2%, FP8 is healthy. If FP8 diverges, your config is wrong.

Activation statistics monitoring

Log activation norms (mean, std, max) per layer per N steps. Trends:

Stable: healthy.
Slowly growing: potential FP8 calibration drift. Update scales more aggressively.
Sudden spike: input distribution shift or numerical instability.

Resume verification

After resuming from checkpoint, validate that the next 10 steps' loss matches what would have been continued from. Catches checkpoint corruption or precision changes silently breaking things.

A/B testing of optimizations

Don't enable FP8 + new optimizer + new architecture all at once. Change one thing at a time, validate each.

Common mistakes

Things teams get wrong with mixed precision.

Skipping calibration

--kv-cache-dtype fp8_e4m3 without --calculate-kv-scales (for inference). Or training without proper FP8 recipe.

Result: large quality regression. Always calibrate.

Wrong format choice

Using e5m2 forward instead of e4m3. e5m2's wider range is for gradients, not activations.

Result: precision loss in forward pass. Use HYBRID (e4m3 forward, e5m2 backward).

FP16 when BF16 would do

FP16 needs loss scaling. BF16 doesn't. Stop using FP16 unless your hardware doesn't support BF16.

Insufficient gradient clipping

Mixed precision is more sensitive to gradient explosions. Use clip_norm = 1.0 minimum.

Optimizer state in BF16

Adam state needs FP32 precision. Storing in BF16 causes optimization to drift.

Result: training quality degrades over many steps.

Mixed BF16 + FP16 in same job

Confusing. Pick one. BF16 for new code; FP16 only for legacy hardware.

Not validating long-term stability

FP8 issues sometimes appear only after thousands of steps. Run a 10k+ step validation before committing.

Future of training precision

Where this is going.

FP4 standardization

By 2027, FP4 forward pass likely standard for frontier training. Quality data is closing the gap with FP8.

NVIDIA Rubin generation will have native FP4 + new MXFP6 for backward.

Block-wise scaling

Per-tensor → per-channel → per-block. Each step recovers quality at lower bit widths.

MXFP4 (block-wise FP4) with 32-element blocks shows promising quality recovery.

Specialized formats per layer

Different layers may use different precisions. Attention QKV in BF16 (sensitive); MLP in FP4 (compute-bound).

Architectural co-design for precision is emerging research.

Mixed-format pipelines

Forward in FP4, backward in FP6, optimizer in FP32. Each phase optimized for its task.

Quantization-aware base model training

Train models from scratch with explicit quantization awareness. Output: model that quantizes well at deployment time without separate QAT.

Some 2025-2026 frontier models likely include this implicitly. Becoming standard practice.

Beyond Adam

Optimizers designed for mixed precision (Lion, Sophia variants) may emerge as standard. Some show better convergence at lower precisions than Adam.

Performance benchmarks

Real numbers from production training.

Training throughput by precision (Llama-3 70B, 8× H100)

Precision	Tokens/sec/GPU	Memory used per GPU
FP32	~800	OOM (doesn't fit)
BF16 mixed	~3,200	~62 GB
FP8 e4m3/e5m2	~5,800	~50 GB

For 70B-class training on H100, FP8 is roughly 1.8× faster than BF16. Cumulative effect over a long training run is significant.

Training throughput by precision (Llama-3 70B, 8× B200)

Precision	Tokens/sec/GPU	Notes
BF16 mixed	~6,400	2× H100 baseline
FP8 mixed	~11,500	1.8× B200 BF16
FP4 mixed	~22,000	early data, quality not yet validated

B200 with FP4 gives roughly 7× the throughput of H100 with BF16 — if quality is acceptable.

Inference throughput by precision (Llama-3 70B, 2× H100)

Precision	Tokens/sec aggregate	Quality
BF16 weights, BF16 KV	800	baseline
FP8 weights, FP8 KV	1500	-0.1 to -0.5 pts
FP8 weights, INT4 KV	1700	mixed
AWQ INT4 weights, FP8 KV	1900	-0.5 to -1 pt
AWQ INT4 weights, INT4 KV	2200	-1 to -2 pts

Choose based on quality tolerance. FP8/FP8 is the safe modern default.

Quality regression by precision (RULER 32k retrieval)

Precision	RULER 32k score (Llama-3 70B)	Drop vs BF16
BF16 baseline	78.5	—
FP8 e4m3 (calibrated)	78.0	0.5
INT8 W8A16	77.8	0.7
INT4 AWQ + INT4 KV	73.2	5.3

INT4 KV breaks long-context retrieval. Be aware of workload sensitivity.

When to use which format: decision matrix

Scenario	Recommended	Reason
Frontier LLM pre-training	FP8 mixed	Best throughput-per-quality
Production fine-tuning	BF16 mixed	Simple, safe
Memory-constrained training	FP8 + INT4 weights	Maximum compression
Inference, chat	FP8 weights, FP8 KV	Modern default
Inference, long-context	BF16 weights, FP8 KV	Quality preservation
Inference, edge	INT4 (AWQ)	Memory critical
Research experiments	BF16 mixed	Simplicity
Ampere/older hardware	INT8 mixed	Fallback
Blackwell production	FP4 (carefully)	Maximum throughput

For most teams, BF16 → FP8 is the upgrade path. Don't skip BF16 thinking you'll go straight to FP4.

Calibration set selection

The data used to compute scaling factors. Matters more than people realize.

Properties of good calibration data

Representative: matches production distribution.
Diverse: covers the input space.
Sufficient: 100-1000 batches typical.
Up-to-date: reflects current data, not 6 months ago.

Bad calibration data symptoms

Quality drop > 1 point on standard benchmarks.
Activation distributions during inference look different from calibration.
NaN/Inf occasionally.

Calibration data sources

Production traces: ideal but may have privacy concerns.
Curated subset of training data: safe baseline.
Synthetic data: convenient but representativeness questionable.
Public benchmarks: easy but may not match your distribution.

Calibration frequency

One-time at deployment: standard for stable workloads.
Periodic (monthly): if traffic distribution shifts.
Continuous (delayed scaling at inference): tracks changes automatically.

For most production: one-time calibration at deployment with optional delayed scaling for runtime adjustment.

Mixed precision in inference vs training

Different concerns.

Training

Goal: throughput + quality.
Forward: FP8 (or BF16).
Backward: FP8 e5m2 (wider range for gradients).
Optimizer: FP32 (stability).
Master weights: FP32 (precision for updates).
Calibration: dynamic (delayed scaling).

Inference

Goal: throughput + memory + quality.
Weights: FP8/INT4 (one-time, static).
KV cache: FP8/INT4.
Activations: BF16 (numerically robust during forward).
Calibration: static at deployment.

Why training needs more

Training's backward pass and optimization steps need wider dynamic range and more precision. Inference is simpler — just forward.

Common mistake: applying training recipe to inference

Training recipes (BF16 forward + FP32 master) are wasteful for inference. Use proper inference quantization (FP8 or INT4 weights) for deployment.

How to validate a mixed-precision pipeline

Steps for due diligence on a new mixed-precision setup.

Phase 1: Sanity check

Train a small model (1-7B) with the new precision. Compare loss curve to BF16 baseline. Should be within 1%.

Phase 2: Quality eval

After training, evaluate on standard benchmarks (MMLU, RULER, HumanEval). Compare to BF16 baseline.

Quality drop > 0.5 points on MMLU = problem.

Phase 3: Long-run stability

Train for 100k+ steps. Watch for:

Loss divergence.
NaN/Inf events.
Quality degradation over time.

Phase 4: Ablation

Try variants. Different calibration sets. Different formats. Find the best for your workload.

Phase 5: Production validation

Deploy to a fraction of traffic. Compare metrics to baseline.

If quality and performance are good: ramp to 100%.

If issues: roll back, debug, retry.

This pipeline catches most mixed-precision problems before they hit production.

Glossary

AMP: Automatic Mixed Precision. PyTorch's library for casting between formats.
autocast: PyTorch context manager that enables automatic mixed-precision.
BF16: Brain Float 16. 1 sign + 8 exp + 7 mantissa.
calibration: process of computing scales for low-precision tensors.
DelayedScaling: TE's strategy for stable FP8 scaling using historical maxima.
dynamic loss scaling: adapt loss scale based on gradient overflow events.
e4m3 / e5m2: FP8 variants.
FP4 / FP8 / FP16 / BF16 / TF32: floating-point formats.
gradient overflow: gradient values exceeding the format's max.
gradient underflow: gradient values rounding to zero.
loss scaling: multiplying loss by a factor to scale up gradients before backward.
master weights: FP32 copy of weights used by the optimizer.
per-tensor / per-channel / per-token scaling: scale granularity options.
Transformer Engine (TE): NVIDIA's mixed-precision library.

References

Foundational papers

Mixed Precision Training — Micikevicius et al., 2017. "Mixed Precision Training." arXiv:1710.03740. ICLR 2018. FP16 + loss scaling — the recipe every later format inherits from.
FP8 Formats for Deep Learning — Micikevicius et al., 2022. "FP8 Formats for Deep Learning." arXiv:2209.05433. The e4m3/e5m2 specification adopted by NVIDIA, Arm, and Intel.
FP8-LM / MS-AMP — Peng et al., 2023. "FP8-LM: Training FP8 Large Language Models." arXiv:2310.18313. Microsoft's framework for FP8 weights, gradients, and optimizer state.
Kalamkar et al., 2019 — "A Study of BFLOAT16 for Deep Learning Training." arXiv:1905.12322. The case for BF16 as the default training format.

Production FP8 training and systems

DeepSeek-V3 — DeepSeek-AI, 2024. "DeepSeek-V3 Technical Report." arXiv:2412.19437. The first widely-documented frontier-scale model trained end-to-end in FP8.
Megatron-LM — Shoeybi et al., 2019. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." arXiv:1909.08053.
ZeRO — Rajbhandari et al., 2019. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." arXiv:1910.02054.

Hardware and framework documentation

NVIDIA Transformer Engine — docs.nvidia.com/deeplearning/transformer-engine/. The reference FP8 library: delayed scaling, layer wrappers, and framework hooks.
NVIDIA Hopper H100 whitepaper — resources.nvidia.com/en-us-tensor-core. FP8 tensor core architecture.

Background reading

See our companion guides on distributed LLM training, NCCL collectives, AI training networking, checkpoint storage and recovery, and inference-time quantization tradeoffs.

Mixed precision case studies

Real-world deployments and lessons.

Case study 1: Llama-3 training in BF16

Meta's Llama-3 (8B, 70B, 405B) trained primarily in BF16.

Key decisions:

Master weights: FP32.
Computations: BF16.
Optimizer state: FP32.
All-reduce: BF16.

Why not FP8?

Llama-3 development started before FP8 was production-ready.
BF16 is well-understood, predictable.
Cost difference vs FP8 was acceptable.

Lesson: BF16 is still the default choice for many production runs.

Case study 2: H100 FP8 for inference

Many companies run H100 inference in FP8 for capacity:

2x throughput vs BF16.
Quality validated against BF16 baseline.
Per-tensor or per-channel quantization.

Lesson: FP8 inference is mainstream; FP8 training is still emerging.

Case study 3: Mixture-of-experts with FP8

Some MoE training runs use FP8 for experts, BF16 for shared layers:

Experts dominate compute → FP8 saves the most.
Shared layers more sensitive → BF16 keeps stability.

Result: ~1.5x throughput improvement with negligible quality regression.

Lesson: hybrid precision (different precisions for different layers) can be optimal.

Case study 4: FP4 inference on Blackwell

Early Blackwell deployments use FP4:

4x throughput vs FP8.
Requires careful calibration.
Specific operators (dense matmul, attention) only.

Lesson: FP4 is here, but only for specific operators with calibration.

Case study 5: Training instability from FP8 misconfiguration

A team observed loss divergence at step 50k of FP8 training. Investigation:

Specific layer's activations exceeded FP8 e4m3 range.
Per-tensor scaling wasn't aggressive enough.
Switched to per-channel for that layer.

Result: stable training. Lesson: FP8 training requires monitoring activation distributions per layer.

Case study 6: Mixed BF16/FP32 for diffusion models

Stable Diffusion-class models often use:

VAE in FP32 (sensitive).
U-Net in BF16.
Attention in FP16 with care.

Lesson: different model components have different precision needs.

Case study 7: Quantization-aware training (QAT)

For deployment at INT4 or INT8:

Train in BF16 with simulated quantization.
Model learns to be robust to quantization noise.
Final deployment runs at lower precision.

Lesson: training and inference precisions can differ; QAT bridges them.

Mixed precision in research

What's emerging.

FP4 training (active research)

Researchers exploring FP4 for end-to-end training.

Challenges:

Even narrower dynamic range.
More aggressive scaling needed.
Sensitivity to specific operators.

Status: research, not production.

Block-wise floating point

Format where a block of values shares an exponent. E.g., MX (Microscaling) format.

Benefits:

Better dynamic range than per-tensor scaling.
Lower memory than full FP precision.

Status: emerging, some hardware support.

Stochastic rounding

Instead of round-to-nearest, randomly round up or down based on remainder.

Benefits:

Unbiased gradient estimates.
Better training stability at low precision.

Status: implemented in some frameworks.

Logarithmic number formats

Represent values in log space. Better for some operations.

Status: research.

Posit numbers

Alternative to IEEE float. Variable precision based on magnitude.

Status: theoretical, limited hardware support.

Hybrid analog-digital

Some research explores analog computation for matrix multiply with digital conversion.

Status: very early.

Implications for production

Most research isn't yet production-ready. But the trajectory is clear:

Precision is decreasing over time.
Hardware-software co-design is essential.
Each new format requires careful validation.

For practitioners: stick with BF16 / FP8 today. Watch FP4 / MX for tomorrow.

Mixed precision tooling deep dive

Tools that support mixed precision.

NVIDIA Transformer Engine (TE)

The reference implementation for FP8 training.

Components:

Custom kernels for FP8 operations.
Automatic scaling factor management.
Integration with PyTorch and JAX.

API:

from transformer_engine.pytorch import Linear, LayerNormLinear

linear = Linear(d_in, d_out, bias=True)
# Forward pass uses FP8 internally

For most teams: TE is the default for FP8 training.

PyTorch native support

torch.amp (automatic mixed precision):

Handles BF16 and FP16 automatically.
Loss scaling for FP16.
No FP8 native support yet (use TE).

For BF16/FP16: torch.amp is sufficient.

JAX bfloat16

JAX has first-class bfloat16 support:

jax.config.update("jax_enable_x64", False).
jnp.bfloat16 type.

JAX integrates with TE for FP8.

Megatron-LM

Megatron-LM has built-in TE integration:

BF16 by default.
FP8 with --fp8-format hybrid.

Most large training runs use Megatron with TE.

NeMo

NVIDIA NeMo wraps Megatron with higher-level APIs.

Mixed precision configuration via YAML.

Lightning

PyTorch Lightning supports mixed precision via:

precision="bf16-mixed".
precision="16-mixed".
precision="fp8" (with TE).

Convenient for training infrastructure.

Custom kernel libraries

For specific operations:

FlashAttention supports BF16, FP16.
Custom fused kernels (e.g., for layer norm).

These often have precision-specific implementations.

Mixed precision FAQ extension

Mixed precision in real architectures

How to apply mixed precision to specific transformer architectures.

Standard transformer (decoder-only)

Per layer:

LayerNorm: BF16 input, BF16 output, FP32 internal.
Attention QKV projection: FP8 forward, BF16 backward.
Attention softmax: BF16 (FP32 for numerically large logits).
Attention output projection: FP8 forward.
FFN: FP8 forward, FP8 backward.
LM head: BF16 (numerically sensitive).

Master weights: FP32. Optimizer state: FP32.

Result: ~2x throughput vs pure BF16 with negligible quality regression.

Mixture-of-experts (MoE)

Different precisions for different parts:

Router: BF16 (small, sensitive).
Experts: FP8 (large, less sensitive in aggregate).
Shared layers: BF16.

Result: ~1.5x throughput vs pure BF16. Experts dominate compute, so FP8 there matters most.

Multi-modal architectures

Per modality:

Vision tower: BF16 (well-validated).
Audio tower: BF16.
Text encoder/decoder: FP8 where stable.
Cross-modal layers: BF16 (sensitive interactions).

Carefully validate each subcomponent.

Diffusion models

For latent diffusion:

VAE encoder/decoder: FP32 or BF16.
U-Net: BF16, FP8 emerging.
Conditioning networks: BF16.

Diffusion is more sensitive to precision than autoregressive LMs.

State-space models (Mamba, etc.)

Different numerical properties than transformers:

SSM kernels: BF16 (FP32 internal for stability).
Linear layers: FP8 can work.
RMSNorm: BF16/FP32.

Less FP8 experience than transformers.

Hybrid architectures

For Jamba-like (mix of attention + SSM):

Each component as appropriate.
Boundaries can be BF16 for stability.

Mix-and-match based on each component.

RNN / LSTM (legacy)

Generally:

BF16 throughout.
FP32 for cell state.
FP8 less common.

Less optimization effort given limited modern use.

Cross-architecture lessons

Embeddings: usually BF16 or FP32.
Output heads: usually higher precision.
Inner layers: lower precision.
Norms: higher precision internally.

These patterns hold across most architectures.

Mixed precision quality validation

How to verify mixed precision doesn't degrade quality.

Baseline establishment

Train identical model in BF16 (or FP32). This is the quality reference.

For new models: this baseline doesn't exist. Need extra caution.

Loss curve comparison

Compare training loss curves:

Baseline vs FP8.
Should track closely.
Divergence indicates issue.

Plot at log scale to see early-training dynamics.

Eval harness

Run standard evals:

MMLU, GSM8K, HumanEval, etc.
Compare scores within statistical noise.

Eval harnesses (lm-eval-harness) are standard.

Specific behavior tests

Domain-specific tests:

Math reasoning.
Code generation.
Long-context retention.
Safety/alignment.

Different from aggregate evals.

Activation distribution monitoring

Per layer, monitor:

Mean activation magnitude.
Standard deviation.
Tail behavior.

Distributions should be similar to baseline.

Gradient distribution monitoring

Per layer, monitor gradient distributions:

No NaNs.
No extreme values.
Stable over training.

Quality alert thresholds

Eval score regression > 1%: investigate.
Loss curve divergence > 5%: investigate.
Activation outliers: investigate.

Tighter or looser based on stakes.

What if quality regresses

Diagnostic steps:

Identify which layer / component.
Selectively raise precision for that component.
Validate fix.
Document for future.

Iterative improvement.

Long-tail behavior

Aggregate evals can mask:

Rare-but-important behaviors.
Specific user scenarios.
Edge cases.

Need long-tail testing too.

Continuous validation

Throughout training:

Periodic eval runs.
Distribution monitoring.
Compare with baseline at each milestone.

Not one-time validation.

Mixed precision compute hardware support

Hardware support across vendors.

NVIDIA

A100: FP32, FP16, BF16, INT8.
H100: adds FP8 (e4m3, e5m2).
H200: same as H100.
B100/B200: adds FP4.

NVIDIA leads in mixed precision tooling.

AMD

MI250X: FP16, BF16.
MI300X: adds FP8 support.

Catching up; FP8 software stack maturing.

Intel Gaudi

Gaudi 2: BF16, FP8.
Gaudi 3: improved FP8.

Native FP8 support is competitive.

Custom silicon

Google TPUs: bfloat16 native.
Cerebras: own format.
Tenstorrent: BF16, FP8 support.

Each has unique format choices.

Software ecosystem maturity

NVIDIA: most mature for FP8.
Others: catching up.

For most teams: NVIDIA path is easiest today.

Mixed precision in production summary

Putting it all together for production deployments.

The default

For new production deployments today: BF16 is the safe default.

It's well-validated, predictable, and supported everywhere.

When to step down to FP8

Step down when:

Cost matters significantly.
You can do thorough validation.
Hardware supports it.
Engineering capacity available.

When to use FP4

Only for specific operators with calibration. Not whole-model training yet.

When to step up to FP32

Step up for:

Numerically sensitive components.
Master weights in mixed-precision setups.
Optimizer state.

The future

5 years from now: FP8 will be standard, FP4 for specific cases. New formats emerging.

Decision summary

Don't over-engineer. Start with BF16, optimize where it pays off.

Validate every change. Document for posterity.

For most teams: 80% of value is in BF16, 15% in FP8, 5% in cutting-edge.

Mixed precision learning resources

For deeper learning.

Papers

Mixed Precision Training (Micikevicius et al., 2018) — original FP16 paper.
FP8 Formats for Deep Learning (Micikevicius et al., 2022) — FP8 standardization.
FP8-LM: Training FP8 Large Language Models (Peng et al., 2023) — practical FP8.
MX (Microscaling) Formats — block-wise FP.

Documentation

NVIDIA Transformer Engine docs.
PyTorch AMP docs.
JAX bfloat16 docs.
Megatron-LM mixed precision configuration.

Talks

NVIDIA GTC mixed precision talks.
NeurIPS / ICML workshop talks.

Code

TE example notebooks.
Megatron-LM examples.
Open-source training scripts.

Communities

/r/MachineLearning.
ML Twitter (research updates).
Workshop communities at major conferences.

For practitioners: hands-on practice is the best learning.

Mixed precision migration playbook extension

Practical migration scenarios.

Migrating from FP16 to BF16

When teams started in FP16, then BF16 became viable:

BF16 has wider dynamic range (~FP32 range, FP16 precision).
Less need for loss scaling.
More forgiving.

Migration:

Switch precision flag.
Disable loss scaling.
Validate quality.

Generally smooth.

Migrating from BF16 to FP8

More involved:

Update framework (Transformer Engine).
Add scaling factor management.
Validate per-layer.
Tune calibration.
Run extensive eval.

Plan 2-4 weeks for medium models.

Migrating from FP8 to FP4

Active research, not production for most yet. Wait for tooling to mature.

Cross-version migrations

Each framework version may change defaults. Read release notes, validate after upgrades.

Mixed precision benchmark numbers

Concrete benchmark numbers across formats and hardware.

Llama-3 70B inference, H100

Format	Throughput (tok/s)	Memory (GB)	Quality
BF16	100	140	Baseline
FP8 (e4m3)	195	75	-0.1% MMLU
INT8	180	75	-0.3% MMLU
INT4	350	40	-1.5% MMLU

FP8 is the sweet spot for most production.

Llama-3 70B training, 1024 H100s

Format	TFLOPS/GPU	MFU	Time/epoch
BF16	350	35%	Baseline
FP8 (mixed)	580	50%	0.6x

FP8 training is ~1.7x faster.

MoE inference (Mixtral 8x22B), H100

Format	Throughput	Memory
BF16	60 tok/s	280 GB
FP8	115 tok/s	145 GB

Routing layer in BF16, experts in FP8.

Diffusion models (SDXL), L40

Format	Steps/sec	Quality
FP32	1.2	Baseline
BF16	2.5	Identical
FP8 (with care)	4.0	Similar

Diffusion benefits significantly from BF16. FP8 requires more care.

Embedding models, A100

Format	Throughput	Memory
FP16	10k qps	Standard
INT8	18k qps	-50%

Embeddings can be aggressively quantized.

These numbers are representative; real numbers depend on configuration.

Mixed precision in agent/RAG systems

How mixed precision affects more complex AI systems.

Agent reasoning

Agents perform multi-step reasoning. Each step involves an LM call.

Precision considerations:

Per-call precision matters less.
Reasoning chain accumulates errors.
Long contexts benefit from lower precision (more memory).

For most agents: BF16 or FP8 fine.

RAG (retrieval-augmented generation)

RAG involves:

Embedding generation.
Retrieval.
LLM inference.

Each can use different precision:

Embeddings: typically FP16 or quantized.
LLM: FP8 or BF16.
Retrieval: lower precision works.

Cost-effective overall.

Tool-using agents

For tool calls:

Structured output sensitivity matters.
Precision affects parsing reliability.

Validate that lower precision doesn't break tool use.

Multi-modal agents

Each modality has its own precision considerations:

Combine with care.
Validate end-to-end.

Long-running agents

Over many calls:

Quality issues accumulate.
Need especially robust validation.

Continuous monitoring is essential.

Per-format throughput math

The reason teams care about FP8 and FP4 reduces to specific throughput numbers on specific hardware. The math.

FLOPs per chip per format

Format	H100 (TFLOPs)	H200 (TFLOPs)	B200 (TFLOPs)	Speedup vs FP32
FP32	67 (vector)	67	80	1.0×
TF32	989	989	1500	14.8×
BF16	1979	1979	4500	29.5×
FP16	1979	1979	4500	29.5×
FP8	3958	3958	9000	59.1×
FP4	n/a	n/a	18000	269× (B200 only)

The math is straightforward — every halving of precision doubles throughput. The catch is that real-world sustained throughput rarely matches peak. Production frontier training in FP8 lands at 40-55% of peak FP8 TFLOPs (sustained). BF16 hits 35-50%. FP4 on Blackwell is too new to have stable production numbers but early reports suggest 30-45%.

Memory savings

Halving precision halves the bytes for the tensors stored in that format. For weights and activations, this directly grows the largest batch you can fit:

FP32 → BF16: 2× larger batch.
BF16 → FP8: another 2× larger batch.
FP8 → FP4: another 2× larger batch.

Optimizer states stay in FP32 typically, so the savings compound less than naively expected. Practical: BF16 → FP8 transition saves ~30% of total training memory (because optimizer state dominates and stays FP32). FP8 → FP4 saves another ~20%.

What you actually save in production

A 70B training run on 32× H100 in BF16 takes ~12 days. Same run in FP8 takes ~7 days. Same run in FP4 on B200 (if it worked at scale) would take ~3 days. The compounding speedup is real but not pure; engineering and validation costs eat into theoretical wins.

When mixed precision breaks: a taxonomy

Specific failure modes mapped to root causes and fixes.

Loss curve diverges in first 1000 steps of FP8 training

Almost always loss-scale or initialization. FP8 has 4 exponent bits in e4m3; the dynamic range is small. Activations in early steps can underflow before scaling stabilizes. Fix: use BF16 for the first 1000 steps, switch to FP8 after warm-up. Most production stacks (Transformer Engine, vLLM training) do this automatically.

Loss is OK but eval scores drop 2-3 points

Layer-specific quantization error accumulating in attention. Try: keep attention in BF16, FP8 only on MLP. Or: switch attention from per-tensor to per-channel scaling. Or: per-row scaling on the softmax logits before they enter FP8.

Gradients show occasional NaN late in training

Out-of-range FP8 representations in gradient accumulation. Use e5m2 (more exponent) for gradients, e4m3 (more precision) for activations — this is Transformer Engine's default for a reason. Verify with assert grad_format == "e5m2" in your training scaffolding.

Quality regression on long-context retrieval but not on short-context

Softmax denominators are tiny at long context; FP8 underflows. Specifically check the attention numerator/denominator computation. Solutions: switch attention to BF16, or use FP8 with explicit max-clamping on logits.

Throughput is the same as BF16 despite enabling FP8

The FP8 path isn't actually engaged. Common causes: incompatible matmul dimensions (FP8 GEMMs require dimensions divisible by 16 for some H100 paths, by 32 for B200), CUDA toolkit too old (need 12.4+ for B200 FP8 instructions), Transformer Engine fallback to BF16 due to numerical safety checks. Profile with Nsight Compute to confirm FP8 kernels are running.

Forward pass differs from BF16 reference by >1% on identical input

Per-tensor scaling overshoots on one tensor's actual maximum. Fix: longer scaling calibration window, or switch to per-channel scaling for that layer. Most production frameworks let you specify per-layer policies.

Comparing FP8 implementations: a deep look

The major stacks that implement FP8 training in 2026, with practical differences.

NVIDIA Transformer Engine (TE)

The reference. Implements per-tensor scaling with delayed scaling (one-step lag) for the forward, dynamic scaling for the backward. Supports Hopper and Blackwell. Integrates with Megatron-LM, NeMo, DeepSpeed via wrapper modules.

import transformer_engine.pytorch as te
linear = te.Linear(input_dim, output_dim, fp8=True)

TE is the only FP8 implementation NVIDIA officially supports for production. Most other implementations are either thin wrappers around TE or independent reimplementations.

MS-AMP (Microsoft)

Three FP8 levels (O1, O2, O3) with progressively more aggressive quantization. O1 quantizes only weights; O3 quantizes everything including optimizer state.

from msamp import deepspeed
model, optimizer = deepspeed.initialize(model, ..., msamp_optimization_level="O2")

MS-AMP integrates with DeepSpeed. Less battle-tested than TE; useful for research and for non-NVIDIA-blessed workflows.

DeepSeek's FP8 implementation

Open-published in the DeepSeek-V3 paper. Uses block-wise scaling (128×128 blocks) instead of per-tensor. Trades a bit of throughput for much better numerical stability — they reported training a 671B-parameter MoE entirely in FP8 with no quality regression.

Block-wise FP8 scaling is becoming standard for 100B+ training. Expect Transformer Engine to adopt it (it's been hinted at in roadmap discussions).

torchao's FP8

PyTorch's native FP8 training library. Released 2024. Per-tensor scaling, less feature-rich than TE but cleanly integrated with FSDP-2 and torch.compile.

from torchao.float8 import convert_to_float8_training
convert_to_float8_training(model)

Production maturity is improving fast. For PyTorch-native pipelines without Megatron, torchao is a reasonable choice.

Comparison table

Implementation	Hopper	Blackwell	Block-wise scaling	Framework integration	Production maturity
Transformer Engine	Yes	Yes	No (per-tensor + delayed)	Megatron, NeMo, DeepSpeed	High
MS-AMP	Yes	Yes	No	DeepSpeed	Medium
DeepSeek FP8	Yes	Yes (custom)	Yes	Megatron (forked)	High (in DeepSeek's stack)
torchao	Yes	Yes	No (improving)	PyTorch FSDP, `torch.compile`	Medium
AMD FP8	No	No	n/a	ROCm	Low (early 2026)

For most teams in 2026: use Transformer Engine via NeMo or Megatron. For research: torchao. For sub-Hopper hardware: BF16 only, FP8 isn't supported.

FP4 training in production

FP4 went from experimental to production-tentative during 2025. The state of the art in mid-2026.

Format details

e2m1 (2 exponent bits, 1 mantissa bit) plus sign. Dynamic range: roughly ~10⁻¹ to ~10¹. 14 distinct values total. The narrow range means scaling is critical — every tensor needs aggressive per-channel or per-block scaling.

What works in FP4 now

Forward pass weights and activations on MLP layers. Attention is still BF16 or FP8 — softmax dynamic range overflows FP4. Optimizer states remain FP32. Master weights remain BF16 or FP8.

What doesn't work yet

Pure FP4 training. Attention layers in FP4. Anything involving rare-feature gradients (where individual rare tokens contribute most of the gradient signal for a parameter). LayerNorm in FP4. Cross-entropy loss in FP4.

Quality data

NVIDIA's published FP4 training papers report ~0.5-1.0 point MMLU regression versus FP8 on Llama-3-class models with full calibration. Real production runs are mixed — DeepSeek's openly published numbers are good; some labs report 2-3 point regressions on harder tasks.

When to use FP4 in 2026

Almost never for from-scratch training of foundation models — the risk isn't worth the speedup. Useful for: (1) inference (well-established), (2) fine-tuning on stable base models (less risk), (3) experimentation and ablation studies. Expect FP4 to mature into the default forward-pass precision by 2027 on Blackwell hardware.

Mixed precision and distributed parallelism

Precision choice interacts with parallelism strategy in non-obvious ways.

TP and per-rank scaling

In tensor parallelism, weights are split across TP ranks. If you're using per-tensor scaling, each rank's scaling factor differs. The all-reduce after attention/MLP mixes tensors with different scales — care is needed to avoid quality regression.

Transformer Engine handles this transparently via per-rank scaling factors that are broadcast as part of the collective. Custom implementations need to handle this manually.

FSDP and parameter sharding precision

FSDP shards parameters across DP ranks. Each rank holds only its shard, gathered on-demand for forward/backward. The all-gather can happen in BF16 or FP8 — the latter halves communication bandwidth.

FSDP-2 supports FP8 all-gather since PyTorch 2.4. Practical: 30-40% reduction in inter-node bandwidth requirement. Worth enabling on bandwidth-bound clusters.

PP and per-stage precision

Different pipeline stages can use different precision. Some training setups use higher precision (BF16) on early and late layers (which are more numerically sensitive) and FP8 on middle layers. Megatron supports per-stage precision configuration.

EP and FP8 routing

For MoE, the all-to-all expert routing can run in FP8, halving bisection-bandwidth pressure. Quality cost: negligible if routers (the gating networks) stay in BF16. DeepSeek-V3 used FP8 all-to-all in production.

Extended FAQ

Q: How much does FP8 actually save on a 70B training run?

End-to-end: 30-45% wall-clock reduction versus BF16, after accounting for FP8 setup overhead, occasional fallback to BF16 for numerically sensitive ops, and validation overhead. A 12-day BF16 run becomes a 7-8 day FP8 run on the same hardware.

Q: Is FP8 deterministic?

Less so than BF16. Per-tensor scaling factors are computed dynamically and can vary across runs due to floating-point ordering in the AllReduce that computes them. Bit-deterministic FP8 training requires fixing the scaling factor schedule, which means slightly worse quality. Most production FP8 training accepts non-determinism.

Q: When should I keep attention in BF16 even if I'm using FP8 elsewhere?

Three cases. (1) Long context (>16K tokens) — softmax denominators get tiny, FP8 underflows. (2) Retrieval-heavy workloads where attention patterns matter precisely. (3) Reasoning models where chain-of-thought quality is sensitive to attention precision. For chat and standard pre-training, FP8 attention works fine.

Q: What's the difference between e4m3 and e5m2?

e4m3: 4 exponent bits, 3 mantissa bits. More precision, less dynamic range. Used for forward activations. e5m2: 5 exponent bits, 2 mantissa bits. Less precision, more dynamic range. Used for backward gradients. Transformer Engine uses e4m3 forward + e5m2 backward by default — this matches the FP8 standard (Micikevicius et al. 2022).

Q: How does FP8 quality compare to INT8 for inference?

FP8 typically beats INT8 by 0.1-0.3 points on quality benchmarks because the floating-point representation handles outlier activations better. INT8 requires more aggressive outlier handling (SmoothQuant, AWQ) to match. For inference, both are viable; production stacks are increasingly defaulting to FP8 for Hopper+ hardware.

Q: Can I train in FP8 without Transformer Engine?

Yes, via torchao, MS-AMP, or custom implementations. TE is the most-tested but not the only option. For research or pre-Hopper hardware where TE doesn't apply, alternatives are fine. For production training at scale, TE is the safest.

Q: What's the role of master weights in mixed precision?

The optimizer updates a master copy of weights in FP32 (or sometimes BF16). The forward pass uses a lower-precision (BF16 or FP8) copy derived from the master. This separates "the precision optimizer math needs" from "the precision matmul uses." Without master weights, accumulated optimizer errors over thousands of steps destroy training.

Q: Does mixed precision affect convergence rate?

Slightly. Lower precision adds noise to gradients, which can either help (regularization) or hurt (slowdown). For BF16: convergence rate identical to FP32. For FP8: typically within 5% of BF16, sometimes faster due to noise injection. For FP4: still being characterized; early data suggests 10-20% more steps needed for equivalent quality.

Q: How do I choose between BF16 and FP16 for legacy hardware?

BF16 unless your hardware doesn't support it. FP16 needs loss scaling and is brittle at long context. BF16's wider exponent matches FP32's dynamic range, eliminating the entire class of overflow bugs. Volta (V100) is the only common GPU without BF16; for V100 use FP16+loss-scaling. Everything Ampere and newer has BF16.

Q: What's per-block scaling and when is it better than per-tensor?

Per-tensor scaling: one scale factor per tensor. Simple but throws away precision on tensors with high dynamic range. Per-block scaling: scale factors per N×N block (typically 128×128). Captures local dynamic range. ~5-15% better quality for the same average bit-rate at modest compute overhead. Per-channel scaling: one factor per output channel. Sweet spot for many workloads. DeepSeek-V3 used per-block scaling for FP8; this is becoming standard for 100B+ training.

Q: How do I detect numerical issues in mixed precision training?

Monitor: gradient norm per layer (sudden spikes indicate underflow), loss curve smoothness (jagged drops suggest precision issues), per-tensor max values (saturating at FP8 max indicates overflow), eval scores per epoch (gradual regression suggests accumulating error). Most production training stacks log all of these.

Q: Will FP4 replace FP8 as the default forward precision?

Probably by 2027-2028 on Blackwell-class hardware. The trajectory: FP8 became default on Hopper in 2024-2025; FP4 will follow on Blackwell. But the format-versus-quality calibration is harder for FP4 — expect more sophisticated scaling schemes and per-layer policy customization to become standard.

Q: What about training in INT8 / INT4?

Done in some niche cases (mobile fine-tuning, on-device training). Quality regression is significant; production foundation training avoids integer formats for now. INT formats are inference-only in mainstream practice.

Q: How does mixed precision interact with `torch.compile`?

torch.compile with FP8 sometimes triggers recompilation due to shape or scale changes. PyTorch 2.5+ handles this better. For TE + torch.compile, expect 5-15% additional throughput on top of FP8's base speedup. Run a hundred-step canary before committing to the combination for a multi-week run.

Q: What's the recommended approach for AMD MI300X in 2026?

BF16. ROCm's FP8 path is improving but lags Hopper's. AMD's MI355X (late 2025) has hardware FP8 support, but software ecosystem (Megatron equivalents, robust calibration) isn't there yet. Use BF16 for MI300X training in 2026; revisit FP8 in 2027.

Q: How do I migrate a BF16-trained model to FP8 inference?

Post-training quantization with calibration. Use a representative dataset (1000-10000 samples). Run AWQ, LLM-Compressor, or NVIDIA's quantization tools. Validate on your evaluation set; expect 0.1-0.3 point quality regression for chat workloads, more for code/math.

Q: What's the relationship between FP8 training and FP8 inference?

A model trained in FP8 has scaling factors baked in. Inference in FP8 reuses these factors, which is more accurate than post-training quantization. The win: serving a model trained in FP8 in FP8 typically gives zero quality regression versus BF16 serving; serving a BF16-trained model in FP8 gives 0.2-0.5 points regression. For production, train and serve in the same precision when possible.

Q: Are there published recipes for training a 70B in FP8?

Yes. NVIDIA's NeMo training recipes for Llama-3 70B include FP8 configurations. The DeepSeek-V3 paper documents their FP8 setup in detail. Meta's Llama-3 405B technical report describes their FP8 + BF16 hybrid. These are starting points; adjust for your data and architecture.

Q: What's "delayed scaling" in FP8 training?

Computing the next step's scale factor from this step's max (one-step lag). Avoids a sync within the forward pass. Slight precision cost (~0.05 points typically) versus immediate scaling. Standard in TE for performance reasons; most production runs use delayed.

Q: How do I validate that FP8 training is producing equivalent quality to BF16?

Run twin training jobs (BF16 reference + FP8 candidate) for 5,000-10,000 steps with identical seeds and data. Compare: loss curves (should be within 5% throughout), eval scores at endpoints (within 0.3 points on MMLU-class benchmarks), per-layer activation statistics (max values, norms). If all match within tolerance, FP8 is safe for the full run.

Q: What's the cost of running these validation experiments?

5-10% of total training compute. For a 90-day frontier training run, that's 4-9 days of validation. Frontier labs do this; mid-tier teams sometimes skip and find out the hard way. Always budget for it.

Table of contents

Key takeaways

Mental model: mixed precision in one minute

The precision formats

Why mixed, not pure

Loss scaling and the FP16 era

BF16: the safe default

When BF16 is right

When BF16 isn't enough

Configuring BF16

FP8: the modern frontier

Two FP8 formats

Calibration and scaling

When FP8 wins

When FP8 hurts

Layer-specific FP8

FP8 quality cost

FP4: emerging

NVIDIA Transformer Engine

Framework support

Auditing a mixed-precision run

Loss curve

Gradient norms

Activation statistics

Loss scaling history

Compare to BF16 baseline

Common failure modes

NaN/Inf in gradients

Loss diverges after ~10000 steps

Quality regression vs BF16 baseline

Slower than expected throughput

Bad scaling on a specific layer

Worked example: switching a training run to FP8

Step 1: baseline

Step 2: enable FP8 with default recipe

Step 3: validate

Step 4: scale up

Step 5: long-run validation

A short history of mixed-precision training

Per-format quality benchmarks

Numerical stability deep dive

Underflow

Overflow

Loss spikes

NaN propagation

Calibration deep dive

Static calibration

Dynamic calibration (delayed scaling)

Per-channel vs per-tensor

Block-wise calibration (FP4)

Calibration data choice

Mixed precision in distributed training

TP and FP8 communication

FSDP and FP8 weights

Gradient accumulation in mixed precision

Loss aggregation across ranks

Per-precision deep dive: every format you might use

FP32 (IEEE 754 single-precision)

TF32 (NVIDIA TensorFloat-32)

BF16 (bfloat16)

FP16 (IEEE 754 half-precision)

FP8 E4M3

FP8 E5M2

FP4 NVFP4

FP4 MXFP4

INT8

INT4 and lower

The format choice matrix

Transformer Engine internals: scaling, recipes, and gotchas

Per-tensor scaling

Delayed scaling (the default recipe)

History-based scaling

Per-row vs per-tensor scaling

The HYBRID recipe

Skipping the BAcc accumulator promotion

FP16 master weights vs FP32 master weights

When Transformer Engine fights you

DeepSeek FP8 training recipe in detail

Block-wise scaling at 128 granularity

Mixed-precision accumulation