Collective Communication for AI Training: NCCL, RCCL, MPI, oneCCL, Gloo — The Complete Guide

Q: How does NCCL handle GPU failures?

NCCL doesn't auto-recover from GPU failures. It hangs, waiting for the failed GPU. Application must detect, tear down, and restart.

Q: Can I use NCCL across multiple data centers?

Theoretically yes, but cross-DC latencies make it impractical. Use federated training instead.

Q: How does NCCL compare to MPI for GPU workloads?

NCCL is significantly faster for GPU-to-GPU collectives. Use NCCL for ML, MPI for traditional HPC.

Q: Should I use NCCL for inference?

Yes for tensor-parallel inference. NCCL handles the all-reduce across GPUs serving a single model.

Q: How much does NCCL_DEBUG=INFO slow things down?

Minimal at INFO level. Don't use TRACE in production.

Q: Why does NCCL have so many environment variables?

Because GPU clusters vary enormously. Defaults work for 80% of cases; tuning for the other 20%.

Q: Is NCCL Open Source?

Yes, but development is NVIDIA-led. Source on GitHub.

Q: How do I report NCCL bugs?

GitHub issues on the NCCL repo. NVIDIA is responsive.

Q: Will NCCL work on AMD GPUs?

No. Use RCCL (AMD's equivalent) for AMD.

Q: How do I integrate NCCL with my own application?

Direct C/C++ API, Python via PyTorch / JAX, etc. PyTorch is the easiest path.

Every modern AI cluster — frontier training, 70B fine-tunes, MoE inference — lives or dies on collective communication. The same primitives (AllReduce, AllGather, ReduceScatter, Broadcast, AlltoAll) show up under every framework: PyTorch DDP and FSDP, Megatron's tensor parallelism, DeepSpeed ZeRO, JAX pjit, vLLM TP. The library implementing them is what stands between you and a 2× slowdown on the same hardware.

This guide covers the libraries you actually encounter — NCCL (NVIDIA, de facto standard), RCCL (AMD's NCCL-compatible drop-in for MI300X/MI325X), oneCCL (Intel, Gaudi and Xeon), MPI (classical HPC), Gloo (PyTorch CPU fallback), SHARP (in-network reductions on NVIDIA Quantum InfiniBand), plus PyTorch c10d and JAX/XLA TPU collectives — then the algorithms (Ring, Tree, CollNet, Double Binary Tree, NVLS), the cross-vendor reality, and the deep NCCL tuning, debugging, and playbooks most readers come for.

Learn NCCL first. Know the others exist for the moment you touch AMD, Intel, TPUs, or CPU clusters. Pair with distributed LLM training, mixed-precision training, NVLink and rack-scale topology, and AI training networking. New to the fundamentals underneath all this? Start with what a GPU is and why AI needs them and the difference between training and inference.

Key takeaways
Mental model: collective communication in one minute
The collective communication landscape in 2026
Library comparison table
Collective primitives explained
Algorithm choices: Ring vs Tree vs CollNet vs Double Binary Tree
Cross-vendor reality: NCCL vs RCCL vs MPI in 2026
What NCCL actually does
Algorithms: Ring, Tree, NVLS
Protocols: LL, LL128, Simple
Topology and how NCCL picks paths
Environment variables that matter
InfiniBand and RoCE
Multi-node training topologies
Debugging hangs
Debugging slow collectives
nccl-tests: the diagnostic tool
Common pathologies and fixes
Algorithm bandwidth math
When to override NCCL defaults
NCCL vs UCX vs libfabric
Tuning for specific frameworks
NCCL determinism and reproducibility
NCCL versus TPU collectives
Advanced env vars
NCCL version-by-version feature timeline
NCCL on AWS EFA SRD: the production reality
NCCL on Slingshot 11 / Cray clusters
NCCL_ALGO and NCCL_PROTO override patterns
Inter-DC NCCL: DiLoCo and DiPaCo style training
NCCL profiling and observability
The bottom line
FAQ
Extended FAQ
Glossary
References

Key takeaways

NCCL handles GPU-to-GPU collectives. Three things to know:

NCCL picks algorithm + protocol + transport automatically based on topology and message size. Defaults are usually right. When wrong, override via env vars.
The two algorithms that matter: Ring for large messages (bandwidth-optimal), Tree for small messages (latency-optimal). NCCL picks based on message size and number of ranks.
The two diagnostics that matter: NCCL_DEBUG=INFO shows which path NCCL chose. nccl-tests (all_reduce_perf) measures actual achieved bandwidth. Run both before assuming anything.

The most common pathologies:

TP all-reduce slow → check that NVLink is actually being used (NCCL_P2P_DISABLE shouldn't be set).
DP all-reduce slow → check IB/RoCE configuration; missing GID or bad QoS will halve bandwidth.
Hangs → mismatched ranks or NCCL version mismatches; NCCL_TIMEOUT env var to bound.
Random slowness → topology mis-detection; explicit NCCL_TOPO_FILE if needed.

Mental model: collective communication in one minute

The named problem is the bandwidth bottleneck of synchronous SGD: every training step, every data-parallel rank must agree on the same averaged gradient before it can take the next optimizer step. Gradients are the size of the model. On a 70B model in bf16 that is 140 GB moved across the fabric, every step, by every worker — and the optimizer cannot start until the slowest rank's last byte arrives. Naively, doubling GPUs doubles communication and the step time stops improving. Communication overhead is what stalls scaling, not flops.

The core idea behind a collective library is to never send the same byte twice. A ring all-reduce arranges N GPUs in a logical ring, splits each gradient tensor into N chunks, and rotates chunks around the ring: in N-1 steps every chunk has been reduced, in another N-1 steps every chunk has been broadcast back. Each link carries roughly 2 * (N-1)/N * size bytes — independent of N, the per-link work is constant. That is the trick. The library's job is to pick the right ring (or tree, or in-switch reduction), the right chunk size, and the right transport (NVLink, IB, RoCE) for the message size and topology you happen to have.

Aspect	Naive parameter-server all-reduce	NCCL ring/tree all-reduce
Bytes per GPU per step	`2 * size` (send + receive)	`2 * (N-1)/N * size` (~`2 * size`)
Bottleneck link	PS uplink, scales as N	Slowest ring link, constant
Latency at 1k GPUs	Linear in N	Log N (tree) or 2(N-1) hops (ring)
Throughput	Capped by PS NIC	~85-95% of link bandwidth
Topology awareness	None	NVLink, NVSwitch, IB rails, SHARP
When it pays off	Toy clusters only	Anything beyond 2 nodes

Conceptually:

# Naive: every rank sends full tensor to rank 0, then receives the sum.
send_to(0, grad); grad = recv_from(0)        # O(N) at rank 0

# NCCL: one call, the library picks ring/tree/NVLS.
dist.all_reduce(grad, op=dist.ReduceOp.SUM)  # ~85% of link BW on 8xH100

One number to remember: a tuned NCCL ring all-reduce on 8xH100 with NVLink hits ~370 GB/s busbw — roughly 85% of the 450 GB/s NVLink ceiling. Anything materially below that on your cluster is a tuning bug.

The rest of this guide is everything that extends or depends on that idea — algorithm selection, protocols, transports, env vars, and the debugging playbook for when the library picks wrong.

The collective communication landscape in 2026

"Collective communication" sounds abstract; in practice it's the half-dozen library calls that synchronize gradients, gather sharded parameters, and route MoE tokens. The library choice is dictated by hardware vendor, framework, and scale. Here is the lay of the land in 2026.

NCCL — NVIDIA Collective Communications Library

The de facto standard for any cluster of NVIDIA GPUs. Closed-source historically but open since 2017; ships in every CUDA toolkit and PyTorch wheel. Optimized for NVLink, NVSwitch, GPUDirect RDMA, InfiniBand, and RoCE. Implements Ring, Tree, CollNet, NVLS (NVLink SHARP), and tuned for Hopper/Blackwell at frontier scale. If you have NVIDIA GPUs, you use NCCL — there is no second choice. Source: NVIDIA's NCCL docs.

RCCL — ROCm Communication Collectives Library

AMD's API-compatible NCCL clone for MI250, MI300X, MI325X, and the upcoming MI400. Built on top of ROCm's HIP runtime. The API is intentionally a drop-in: ncclAllReduce → rcclAllReduce, same flags, same algorithms (Ring, Tree, hierarchical). For most PyTorch users on AMD, torch.distributed with backend nccl actually links to RCCL transparently. Performance on xGMI (AMD's NVLink analog) and Infinity Fabric is competitive with NCCL on equivalent NVIDIA hardware, but the ecosystem (debugging tools, profilers, error messages) lags. Source: github.com/ROCm/rccl.

oneCCL — Intel's Collective Communications Library

Intel's collective library, part of oneAPI. Targets Gaudi 2/3 (Habana), Xeon CPUs, and Ponte Vecchio/Falcon Shores GPUs. Uses Intel's MPI runtime underneath for inter-node and HCCL (Habana's intra-node collective fabric) on Gaudi. Common in academic clusters, Aurora-class supercomputers, and Intel-flavored AI deployments. PyTorch's Habana integration plugs oneCCL in via the hccl backend. Source: github.com/oneapi-src/oneCCL.

MPI — the classical HPC primitive

The original. OpenMPI, MPICH, and NVIDIA-flavored MVAPICH have been doing AllReduce on supercomputers since the 1990s. MPI is still ubiquitous in HPC and Slurm-based clusters as the launcher (mpirun, srun) even when NCCL does the actual work. For pure-CPU jobs, mixed CPU/GPU workflows, or simulation codes with sparse AI components, MPI's collectives are still the right answer — they implement decades-old algorithms (Patarasuk & Yuan's bandwidth-optimal AllReduce, JPDC 2009) that NCCL ultimately re-implements on GPUs. Source: mpi-forum.org.

Gloo — PyTorch's CPU and fallback collective

Originally built by Facebook for distributed training before NCCL was open-sourced. Supports both CPU and GPU tensors, falls back to TCP when RDMA isn't available, and ships as a PyTorch backend (backend="gloo"). It is the default when no NCCL is installed and the only sane choice for CPU-only distributed jobs, debugging ("does my training script even work without IB?"), and mixed CPU+GPU parameter servers. Gloo is slower than NCCL on GPU clusters but more permissive about network conditions. Source: github.com/facebookincubator/gloo.

SHARP — Scalable Hierarchical Aggregation and Reduction Protocol

NVIDIA's (originally Mellanox's) in-network reduction technology. SHARP-capable Quantum InfiniBand switches perform the AllReduce inside the switch fabric instead of at the endpoints — every GPU sends its tensor, the switch tree computes the sum, every GPU receives the result. Cuts AllReduce latency by 2-3× on large clusters and halves the bytes on the wire. NCCL automatically uses SHARP when the fabric supports it (NCCL_COLLNET_ENABLE=1). Critical at 1k+ GPU scale; nice-to-have below that. NVLink SHARP (NVLS) is the in-NVSwitch analog for intra-node. Source: NVIDIA SHARP docs.

PyTorch's c10d — the unified interface

When you write torch.distributed.init_process_group(backend="nccl"), you're invoking PyTorch's c10d (Caffe2 / 10-dimensional distributed) layer. c10d is the abstraction that lets the same Python code dispatch to NCCL, RCCL, Gloo, MPI, oneCCL, or HCCL depending on what's available. It implements the ProcessGroup interface, batches small ops, manages async work handles, and integrates with autograd. Most production PyTorch training never touches NCCL directly — it talks to c10d, which talks to NCCL. Source: pytorch.org/docs/stable/distributed.html.

JAX's pjit / XLA collectives

On TPUs (and increasingly GPUs), JAX bypasses NCCL entirely. XLA lowers jax.pmap, pjit, and shard_map operations to TPU-native collectives that run on the ICI (Inter-Chip Interconnect) torus fabric. The primitives are conceptually identical (psum, all_gather, all_to_all) but the implementation is in the XLA compiler. On GPU backends, XLA still calls NCCL underneath. The TPU-native path is what makes Gemini-scale training tractable on Google's pods.

When each is the right choice

NVIDIA-only cluster → NCCL. Period.
AMD MI300X / MI325X cluster → RCCL via PyTorch's nccl backend (it's actually RCCL).
Intel Gaudi 3 cluster → oneCCL via HCCL backend.
TPU v5p / Trillium pod → JAX + XLA. No choice.
CPU-only or development laptop → Gloo.
Slurm/HPC center with mixed CPU+GPU jobs → MPI as launcher, NCCL/RCCL for GPU collectives.
Frontier-scale (1k+ GPU) InfiniBand cluster → NCCL with SHARP enabled.
You don't know yet → PyTorch c10d will pick something reasonable. Start there.

The dirty truth: 95% of production AI training in 2026 runs on NCCL or RCCL. The other libraries matter, but they matter at the edges.

Library comparison table

Library	Vendor / origin	Primary backend(s)	Scope	Tuning surface	Where it fits
NCCL	NVIDIA	NVLink, NVSwitch, IB, RoCE, GPUDirect	NVIDIA GPU collectives	`NCCL_*` env vars (~60), topo files, SHARP	Every NVIDIA AI cluster, from 2× H100 to 100k× B200
RCCL	AMD	xGMI, Infinity Fabric, IB, RoCE	AMD GPU collectives	`RCCL_*` env vars (NCCL-compatible)	MI250/MI300X/MI325X clusters; transparent under PyTorch
oneCCL	Intel	HCCL (Gaudi), MPI, Xe-Link	Intel GPU + CPU collectives	`CCL_*` env vars, oneAPI knobs	Gaudi 2/3 supercomputers, Aurora, Intel-flavored DL
MPI	MPI Forum (OpenMPI, MPICH, MVAPICH)	TCP, IB verbs, UCX, libfabric	General-purpose HPC, any hardware	MCA params, runtime tuning	HPC sims, CPU clusters, Slurm launcher even when NCCL runs ops
Gloo	Meta (Facebook)	TCP, IB (limited)	CPU + simple GPU collectives	Minimal — backend choice, transport	Dev/debug, CPU-only DDP, mixed CPU+GPU param servers
SHARP	NVIDIA (Mellanox)	InfiniBand Quantum / Quantum-2/3 switches	In-network AllReduce, Broadcast	`NCCL_COLLNET_ENABLE`, SHARP daemon config	Large IB clusters; enabled inside NCCL/RCCL
PyTorch c10d	Meta	Dispatches to NCCL/RCCL/Gloo/MPI	Python-level abstraction	`backend=` flag, env vars passed through	Almost every PyTorch training script
JAX / XLA collectives	Google	TPU ICI, GPU (via NCCL)	Compiler-lowered collectives	XLA flags, sharding annotations	All JAX code; TPU-native at pod scale

The two numbers that actually predict performance are the same for every library on this list: how close the achieved bandwidth gets to the per-link bus bandwidth, and the small-message latency floor. NCCL on a properly-tuned 8× H100 NVSwitch box hits ~750 GB/s out of 900 GB/s theoretical. RCCL on a comparable 8× MI300X box hits ~600 GB/s out of ~896 GB/s of xGMI. The rest of this guide is mostly about closing that gap.

Collective primitives explained: AllReduce, AllGather, ReduceScatter, Broadcast, AlltoAll

Every framework you'll use — PyTorch DDP, FSDP, DeepSpeed ZeRO, Megatron-LM, vLLM, JAX pjit — composes its parallelism from this same handful of collectives. Understanding the primitives makes every layer above them legible.

AllReduce

Every rank contributes a tensor; every rank receives the sum (or other reduction) of all contributions.

Before:                After AllReduce(sum):
rank 0: [1, 2, 3]      rank 0: [10, 14, 18]
rank 1: [4, 5, 6]  →   rank 1: [10, 14, 18]
rank 2: [5, 7, 9]      rank 2: [10, 14, 18]

ML use case: Data-parallel gradient synchronization. After backward, every rank holds the gradient of its mini-batch shard; AllReduce sums them into the full-batch gradient that every rank then uses to update weights. This is the single most expensive collective in any DDP training run. See "Accurate, Large Minibatch SGD" for the canonical analysis.

AllGather

Every rank contributes a chunk; every rank receives the full concatenation.

Before:                After AllGather:
rank 0: [A]            rank 0: [A, B, C]
rank 1: [B]        →   rank 1: [A, B, C]
rank 2: [C]            rank 2: [A, B, C]

ML use case: FSDP / ZeRO-3 parameter gathering. Each rank stores a shard of the model weights; before a layer's forward pass, the framework AllGathers the full layer's weights onto every rank. Also used for tensor-parallel output gathering at the end of column-parallel layers in Megatron-LM.

ReduceScatter

Every rank contributes a full tensor; each rank receives one reduced shard.

Before:                          After ReduceScatter(sum):
rank 0: [g0_0, g0_1, g0_2]       rank 0: [g0_0+g1_0+g2_0]
rank 1: [g1_0, g1_1, g1_2]   →   rank 1: [g0_1+g1_1+g2_1]
rank 2: [g2_0, g2_1, g2_2]       rank 2: [g0_2+g1_2+g2_2]

ML use case: FSDP gradient sharding. After backward, ReduceScatter sums the gradients and gives each rank only the shard it owns — half the bytes of an equivalent AllReduce + scatter sequence. AllReduce = ReduceScatter + AllGather, which is why high-performance implementations use this decomposition under the hood.

Broadcast

One rank sends; all others receive an identical copy.

Before:                After Broadcast(root=0):
rank 0: [X, Y, Z]      rank 0: [X, Y, Z]
rank 1: []         →   rank 1: [X, Y, Z]
rank 2: []             rank 2: [X, Y, Z]

ML use case: Initial parameter distribution at job startup, broadcasting an evaluation result from rank 0, distributing the next batch's RNG seed.

AlltoAll

Every rank sends a different chunk to every other rank.

Before:                          After AlltoAll:
rank 0: [a0, a1, a2]             rank 0: [a0, b0, c0]
rank 1: [b0, b1, b2]         →   rank 1: [a1, b1, c1]
rank 2: [c0, c1, c2]             rank 2: [a2, b2, c2]

ML use case: MoE expert routing. Every token has been routed to an expert; an AlltoAll permutes tokens across ranks so each expert (which lives on one rank) receives only the tokens routed to it. After the expert computation, a second AlltoAll inverts the permutation. This is the collective that gates MoE training throughput — see mixture-of-experts serving and DeepSeek-V3's engineering on collectives at scale.

Send / Recv (point-to-point)

Not technically collectives but they live in the same library. One rank sends, one rank receives.

ML use case: Pipeline parallelism. The output activations of one stage are point-to-point sent to the rank holding the next stage. Used heavily in distributed LLM training and in disaggregated inference to ship KV cache between prefill and decode workers.

How frameworks compose them

PyTorch DDP → AllReduce on gradients, per bucket.
FSDP / ZeRO-3 → AllGather on params (forward), ReduceScatter on grads (backward).
Megatron tensor parallelism → AllReduce on activations (row-parallel), AllGather on outputs (column-parallel).
Megatron pipeline parallelism → Send/Recv between stages.
DeepSpeed MoE → AlltoAll twice per MoE layer.
Inference TP → AllReduce per transformer layer's attention and MLP output projection.

Algorithm choices: Ring vs Tree vs CollNet vs Double Binary Tree

For any given primitive (say AllReduce), the library has multiple algorithms to choose from. The right pick depends on message size, rank count, and topology. NCCL, RCCL, and MPI all implement variants of the same core set; the names differ.

Ring

Ranks form a logical ring. Each rank sends a chunk to the next rank for N-1 steps (reduce-scatter), then another N-1 steps (all-gather). Total bytes per rank: 2·(N-1)/N · M — within a constant factor of the theoretical lower bound from Patarasuk & Yuan (JPDC 2009).

Strength: bandwidth-optimal. Achieves near-link-rate on large messages.
Weakness: O(N) latency. Each step is an RTT.
Best for: large messages (gradients in DDP, ZeRO ReduceScatter), small rank counts (within-node, single-rail).

Tree (Binary / K-ary)

Ranks form a binary tree. Reduction flows up to the root in O(log N) steps, then the result broadcasts back down.

Strength: latency-optimal. Wins on small messages.
Weakness: non-leaf nodes do more work; per-link bandwidth is not saturated.
Best for: small messages (control tensors, eval metrics, MoE gate outputs).

NCCL picks Tree below ~64 KB messages, Ring above. The crossover is tunable via NCCL_TREE_THRESHOLD.

Double Binary Tree

A clever MPI-era improvement (Sanders et al.). Build two binary trees with disjoint leaf sets, and pipeline data across both. Every node is non-leaf in exactly one tree and leaf in the other, so per-link load is balanced.

Strength: roughly 2× the throughput of a single tree at the same latency.
Weakness: more complex; only worth it at moderate-to-large scale.
Best for: medium messages on classical HPC fabrics. OpenMPI and MPICH use this heavily; NCCL has its own variant.

CollNet

NCCL's algorithm that offloads the reduction to the network. In practice this means SHARP on InfiniBand Quantum switches. The endpoint sends its tensor once; the switch tree does the reduction; the endpoint receives the result. No 2·(N-1)/N factor — every byte crosses the wire once.

Strength: halves bytes-on-the-wire vs Ring; reduces latency on large clusters.
Weakness: requires SHARP-capable switches and the SHARP daemon.
Best for: 256+ GPU clusters on NVIDIA Quantum IB.

NVLS (NVLink SHARP)

The intra-node version. NVSwitch chips on Hopper/Blackwell HGX baseboards perform the reduction in the switch silicon itself. Same idea as CollNet, applied to the NVLink fabric.

Strength: halves NVLink traffic for AllReduce on 8× H100 and 8× B200 boxes.
Best for: every modern NVSwitch system. NCCL 2.16+ auto-enables on capable hardware.

Hierarchical (intra-then-inter)

For multi-node jobs, nearly all libraries first reduce within each node (over NVLink), then between nodes (over IB/RoCE), then broadcast back inside each node. This collapses the slow inter-node hop to a single AllReduce among one-rank-per-node instead of gpus_per_node × ranks.

Strength: mandatory at multi-node scale; without it, slow inter-node traffic dominates.
Best for: every multi-node training job. NCCL does this transparently.

How to know what was picked

NCCL_DEBUG=INFO logs the chosen algorithm and protocol on init. For systematic sweeps, use nccl-tests:

./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

This walks message sizes from 8 B to 1 GB and reports achieved bus bandwidth at each size. A flat bandwidth curve across sizes is healthy; a cliff at a specific size usually means an algorithm crossover going wrong.

Cross-vendor reality: NCCL vs RCCL vs MPI in 2026

The honest comparison.

NCCL on NVIDIA

Maturity: highest. 10 years of production deployment, every frontier lab uses it.
Hardware coverage: Volta and newer. Tuned for Hopper and Blackwell, with NVLS, CollNet, GPUDirect RDMA, and SHARP all first-class.
Performance: ~80-90% of peak link bandwidth on properly-configured clusters. At 8× H100 NVSwitch, ~750 GB/s AllReduce bus bandwidth out of 900 GB/s theoretical.
Debugging: good. NCCL_DEBUG=INFO is verbose and useful; profilers like Nsight Systems show NCCL kernels inline with compute.

RCCL on AMD

Maturity: improving fast. The MI300X generation drove serious investment; the MI325X / MI400 generation has closed most gaps.
API compatibility: drop-in for NCCL. PyTorch on ROCm uses backend="nccl" and the linker resolves it to RCCL.
Performance: competitive on equivalent hardware. 8× MI300X with xGMI achieves ~600 GB/s AllReduce out of ~896 GB/s; the ratio is similar to NCCL's.
Gotchas: SHARP equivalent is less mature; debugging tools (rocprof, rocgdb) lag CUDA tooling; some env vars behave differently from their NCCL namesakes. New AMD users should expect a 1-2 week tuning curve they wouldn't face on NVIDIA.

MPI as the universal fallback

When you need it: Slurm-launched HPC jobs where the launcher is MPI (srun --mpi=pmix); pure-CPU distributed training (rare in 2026 but still real for some scientific ML); mixed simulation + training workflows.
NCCL still does the actual GPU collectives. MPI launches the processes and handles process groups; NCCL handles the AllReduce.
OpenMPI vs MPICH vs MVAPICH: OpenMPI is the most common; MVAPICH is NVIDIA-flavored with the deepest GPUDirect integration; MPICH is the academic reference. Choose based on what your cluster admin installed — they're API-compatible at the application level.

The honest decision tree

100% NVIDIA fleet → NCCL.
100% AMD fleet → RCCL.
100% Intel Gaudi fleet → oneCCL.
100% TPU fleet → JAX + XLA.
Mixed-vendor fleet → don't. Or, if forced, treat each vendor as its own pool and use higher-level orchestration (Ray, Kubernetes) to schedule across pools. Mixing NCCL and RCCL inside one collective is not supported.
Pure-CPU or development → Gloo.

For the broader story of how these libraries plug into the training stack, see distributed LLM training and NVLink and rack-scale topology. For how they show up in inference, see LLM serving and mixture-of-experts serving.

A short history of NCCL

NCCL began in 2015 as NVIDIA's internal solution to the multi-GPU communication problem. By 2017 it was open-sourced (the same year Horovod popularized ring-allreduce for TensorFlow — see arXiv:1802.05799). By 2020 it was the de facto standard for GPU-to-GPU collectives in deep learning.

A timeline of key changes:

NCCL 1.x (2015-2016): basic ring all-reduce on a single node. PCIe-based.

NCCL 2.0 (2017): open-sourced. Added multi-node support via Sockets and InfiniBand. Tree algorithm for small messages.

NCCL 2.4 (2019): NVLink-aware routing. Major performance improvements on DGX-1/DGX-2 systems.

NCCL 2.7 (2020): GPUDirect RDMA improvements. SHARP support (in-network reductions on Mellanox switches).

NCCL 2.10 (2021): improved multi-thread support. Better topology detection.

NCCL 2.16 (2022): NVLink SHARP (NVLS). Major win on H100 NVSwitch systems.

NCCL 2.18 (2023): CUDA Graph capture support for collectives. Reduced launch overhead.

NCCL 2.20+ (2024): improvements for very-large-scale training (10k+ GPUs). Better adaptive routing.

The point: NCCL has been continuously improving. Keep up to date — NCCL 2.21+ on Hopper/Blackwell hardware delivers materially better performance than NCCL 2.10 on the same hardware.

What NCCL actually does

NCCL provides a small set of collective primitives:

all_reduce: every rank contributes a tensor; every rank receives the sum (or other reduction). Used for DP gradient synchronization, TP activation reduction.
all_gather: every rank contributes a chunk; every rank receives the full concatenation. Used for FSDP/ZeRO parameter gathering.
reduce_scatter: every rank contributes a tensor; each rank receives one chunk of the reduced result. Used for FSDP gradient sharding.
broadcast: one rank sends, all others receive. Used for parameter initialization.
all_to_all: every rank sends a different chunk to every other rank. Used for MoE expert routing.
send / recv: point-to-point. Used for pipeline parallelism.

For each primitive, NCCL has multiple algorithms. The library picks one based on:

Number of ranks.
Message size.
Topology (NVLink, NVSwitch, IB, RoCE, mixed).
Hardware version (Hopper, Blackwell).

This auto-selection is what makes NCCL "just work" 95% of the time. The other 5% is what this guide is for.

Algorithms: Ring, Tree, NVLS

Ring

Ranks form a logical ring. Data flows around the ring in chunks. Bandwidth-optimal for large messages.

For all_reduce with N ranks and message size M:

Phase 1 (reduce-scatter): each rank sends M/N to its neighbor, N-1 steps.
Phase 2 (all-gather): each rank sends M/N to its neighbor, N-1 steps.
Total bytes per rank: 2(N-1)/N × M, very close to 2M for large N.

Achievable bandwidth approaches the per-link bandwidth. On NVLink-4 with 8 GPUs, ~750 GB/s achievable.

Tree

Hierarchical reduction. Some ranks aggregate from others, then propagate. Latency-optimal for small messages.

For small messages, Ring's many round-trips dominate latency. Tree completes in O(log N) hops instead of O(N), making it faster for small messages.

NCCL automatically picks Tree below ~64 KB messages, Ring above. Crossover threshold is configurable.

NVLS (NVLink SHARP)

A hardware-accelerated variant for Hopper+ NVLink fabrics. NVSwitch's SHARP feature performs in-network reductions. Can deliver near-2× the throughput of Ring for medium messages.

NVLS is automatic on supported hardware (DGX H100, B200) and message sizes. You'll see it in NCCL_DEBUG=INFO output as NVLS algorithm choice.

How NCCL picks

By default: Tree for tiny messages, Ring for large, NVLS for medium messages on supported hardware.

Override:

NCCL_ALGO=Ring,Tree    # Restrict to these algorithms
NCCL_ALGO=NVLS         # Force NVLS (errors if not supported)

Most users don't tune this; the defaults are tuned for the hardware NCCL detects.

Protocols: LL, LL128, Simple

NCCL has three protocols for moving data:

LL (low-latency): small messages, high overhead, lowest possible latency. Used for small all-reduce.
LL128: extended LL for slightly larger messages. NVLink-specific.
Simple: high-bandwidth protocol for large messages.

These layer on top of the algorithm choice. Ring + Simple is the typical large-message path; Tree + LL is the typical small-message path.

Override:

NCCL_PROTO=Simple,LL   # Restrict

Rarely necessary to tune.

Topology and how NCCL picks paths

NCCL detects topology at init time:

Which GPUs are connected via NVLink (and bandwidth).
Which are connected via PCIe (slower).
Which are connected via IB or RoCE (across nodes).

It builds a graph and picks paths. For all_reduce on 8 GPUs in one DGX:

All 8 are NVLink-connected via NVSwitch.
NCCL picks Ring or NVLS depending on message size.

For all_reduce on 8 GPUs across 2 nodes (4 per node):

Within-node NVLink, between-node IB.
NCCL builds a hierarchical reduce: within-node first (fast), between-node second (slower).

Topology hints

If NCCL mis-detects topology, you can provide a hint file:

NCCL_TOPO_FILE=/path/to/topo.xml

The format is documented in NCCL docs. Rarely needed unless your hardware does something unusual.

Verification

NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=GRAPH

Shows the topology NCCL detected and the path it chose. Useful sanity check during initial deployment.

Environment variables that matter

The handful of NCCL env vars that fix 80% of issues:

NCCL_DEBUG

Logging verbosity. Default WARN (silent unless errors).

INFO: shows initialization, topology, algorithm choice. Set this when investigating anything.
TRACE: very verbose. Only for debugging weird issues.

NCCL_DEBUG_SUBSYS

Filter by subsystem.

INIT: initialization only.
GRAPH: topology and path selection.
NET: network transport details.
ALL: everything.

Combine: NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=GRAPH,NET.

NCCL_TIMEOUT

Maximum time to wait for a collective. Default 30 seconds.

For training: set to 1800 (30 min) to bound stalls without false positives.

NCCL_P2P_DISABLE

Disable peer-to-peer between GPUs. Should be 0 (enabled) by default. If accidentally set to 1, NVLink is bypassed and everything goes via host memory — 10× slower.

Always check this if collectives are mysteriously slow.

NCCL_NET_GDR_LEVEL

GPU Direct RDMA (GDR) level. Controls when NCCL uses GPU-to-NIC direct DMA vs going through host memory.

5 (PHB) = always use GDR. Fastest but requires hardware support.
0 = never use GDR. Slowest, fallback for buggy IB drivers.

Default is auto-detect. Setting NCCL_NET_GDR_LEVEL=5 and verifying it works is a common tuning step.

NCCL_IB_HCA

Which IB HCAs (host channel adapters) to use. Multi-NIC nodes need this set explicitly.

NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3

NCCL_IB_GID_INDEX

Which GID (Global ID) to use for RoCE. Mis-configured GID = falls back to slower path. Common cause of "RoCE works but at 1/2 expected bandwidth."

NCCL_BUFFSIZE

Internal buffer size. Default 4 MB. Larger can help bandwidth at the cost of memory; smaller can hurt.

NCCL_NSOCKS_PERTHREAD

Number of TCP sockets per IB rail. Default 2. Higher (4-8) can help with high-throughput IB.

NCCL_SOCKET_IFNAME

For TCP transport (used when IB is unavailable), which network interface. Common: NCCL_SOCKET_IFNAME=eth0,eth1.

Putting it together

A reasonable production env:

export NCCL_DEBUG=WARN
export NCCL_DEBUG_FILE=/var/log/nccl-%h-%p.log
export NCCL_TIMEOUT=1800
export NCCL_IB_GID_INDEX=3       # adjust per RoCE config
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3

InfiniBand and RoCE

NCCL uses two transports for inter-node:

InfiniBand (IB)

NVIDIA's preferred fabric. Native verbs API, low-latency, high-bandwidth.

Mellanox/NVIDIA ConnectX adapters.
200 Gb/s, 400 Gb/s, 800 Gb/s per port (current generation).
Rail-optimized topologies for AI training.

NCCL works well with IB out of the box if drivers and OFED are installed correctly.

RoCE (RDMA over Converged Ethernet)

Ethernet alternative. Same RDMA semantics on Ethernet.

Requires lossless Ethernet (PFC, ECN).
Common in clouds where IB isn't available (some AWS instances, GCP TPU clusters).

RoCE configuration is more error-prone than IB. Common issues:

Wrong GID index.
PFC misconfiguration causing packet loss → catastrophic NCCL slowdowns.
ECN not properly set.

For RoCE, always run nccl-tests/all_reduce_perf and verify achieved bandwidth matches expectations. If you're getting <50% of theoretical, something is misconfigured.

Picking IB or RoCE

If you have a choice:

Greenfield deployment, prioritizing performance: IB.
Cloud-hosted, prioritizing flexibility: RoCE if available.
On-prem, mixed workloads: depends on existing infrastructure.

Most major clouds (AWS, GCP, Azure) offer both depending on instance type. For a deeper IB vs RoCE comparison, see our AI training networking guide.

Multi-node training topologies

Rail-optimized topology

For large training clusters, the canonical pattern:

Each node has 8 GPUs and 8 IB NICs (1 NIC per GPU).
8 IB rails: rail 0 connects all GPU 0s across all nodes, rail 1 all GPU 1s, etc.
Each rail has a dedicated switch.

This minimizes cross-rail contention. NCCL automatically uses rail-optimized routing.

Fat-tree topology

Older pattern: all GPUs connect through a hierarchical fat-tree. Less optimal than rail-optimized but more flexible for mixed workloads.

NVLink islands within fat-tree

Modern setups combine: NVLink within a node (full bisection), IB between nodes. NCCL exploits this hierarchy automatically.

GB200 NVL72: rack-scale NVLink

GB200's 72-GPU rack treats the rack as one NVLink fabric. NCCL can use NVLink across the entire rack — TP=72 within one fabric. Reduces inter-rack IB traffic. See our NVLink and rack-scale topology guide and GPU architecture reference for hardware context.

Debugging hangs

NCCL hangs are common at scale. The pattern: one rank's collective hangs, all ranks block waiting.

First step: enable detailed logging

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_DEBUG_FILE=/var/log/nccl.log

Reproduce the hang. Look at logs.

Common hang causes

Mismatched call counts: rank 0 calls all_reduce 100 times; rank 1 calls 99 times. The 100th call on rank 0 hangs forever.

Fix: structural bug in your code. Add logging at every collective.

NCCL version mismatch across ranks: rank 0 uses NCCL 2.20; rank 1 uses 2.18. They might successfully connect but exhibit subtle bugs.

Fix: pin NCCL version in your container; verify with python -c "import torch; print(torch.cuda.nccl.version())".

Network partition: an IB link goes down mid-collective. The peer never receives the message.

Fix: use NCCL_TIMEOUT to bound. Restart on timeout.

GPU went OOM but didn't crash: rank 1's allocator failed silently; subsequent ops never run.

Fix: enable strict OOM mode in your framework. Crash on first OOM.

Debugging command

# Send SIGUSR1 to a hung process to get NCCL state dump
kill -USR1 $PID
# NCCL will dump current collective state to its debug log

In recent NCCL versions, py-spy can also help: py-spy dump --pid $PID shows the Python stack trace, which often reveals the call site.

Debugging slow collectives

Slow but not hung is more common than outright hangs. Symptoms: training step time is 2× expected; per-step time has high variance.

Step 1: measure achieved bandwidth

Run nccl-tests/all_reduce_perf on your cluster. Compare achieved bandwidth to theoretical:

NVLink-4 (single node): ~750 GB/s achievable on Ring.
IB 400 Gb/s: ~45 GB/s per direction achievable.
IB 200 Gb/s: ~22 GB/s.

If you're getting <50% of expected, something is wrong.

Step 2: check algorithm choice

NCCL_DEBUG=INFO 2>&1 | grep -i "algo\|nvls\|ring\|tree"

Verify NCCL is picking the algorithm you expect. Wrong choice (Tree on large messages, Ring on small) can halve throughput.

Step 3: check P2P

nvidia-smi topo -m

Should show NV# for NVLink-connected GPUs. If only PHB (PCIe), NCCL is using slow paths.

Step 4: check IB

ibv_devinfo  # IB device states
ibstat       # IB link states (LinkUp = good)

Bad GID, link down, or wrong fabric = slow.

Step 5: check for stragglers

If one rank is consistently slower, that rank dictates step time. Profile with NVIDIA Nsight Systems to find which rank is slow and why.

nccl-tests: the diagnostic tool

nccl-tests (https://github.com/NVIDIA/nccl-tests) is the canonical NCCL benchmark.

Installation

git clone https://github.com/NVIDIA/nccl-tests
cd nccl-tests
make MPI=1

Running all_reduce_perf

Single node, 8 GPUs:

./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Multi-node, 16 GPUs:

mpirun -np 16 -hostfile hosts.txt ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Interpreting output

The output has columns: size, time, algbw (algorithmic bandwidth), busbw (bus bandwidth).

For all_reduce, busbw is the metric to watch. Compare to:

NVLink-4 (single node, 8 GPUs): expect ~750 GB/s busbw on 1 GB messages.
IB 400 Gb/s (multi-node): expect ~40 GB/s busbw on 1 GB messages.

If you're well below these, troubleshoot.

Other tests

all_gather_perf, reduce_scatter_perf, broadcast_perf: similar pattern for other collectives.
alltoall_perf: for MoE workloads. Tests bisection bandwidth.

Common pathologies and fixes

Pathology 1: TP slow, decode tokens/sec lower than expected

Cause: NCCL not using NVLink. Often due to NCCL_P2P_DISABLE=1 or topology mis-detection.

Fix: unset NCCL_P2P_DISABLE, verify with nvidia-smi topo -m showing NV# connections.

Pathology 2: DP all-reduce slow across nodes

Cause: GDR not enabled, or wrong IB GID, or PFC not configured for RoCE.

Fix:

export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_GID_INDEX=3   # check correct value for your fabric

Verify with nccl-tests.

Pathology 3: random hangs in long training runs

Cause: cosmic-ray bit flips, transient network blips, GPU thermal throttling causing one rank to fall behind.

Fix:

export NCCL_TIMEOUT=1800

Plus framework-level resume-from-checkpoint.

Pathology 4: NCCL warning "no GPU Direct RDMA"

Cause: PCIe topology between GPU and NIC isn't compatible with GDR. Common on consumer hardware or improperly-cabled servers.

Fix: cable GPUs to NICs through the same PCIe switch. Or accept the slower path.

Pathology 5: "NCCL WARN Couldn't bind socket to specified IB device"

Cause: trying to use specific HCAs that don't exist or are misconfigured.

Fix: unset NCCL_IB_HCA to let NCCL auto-discover, then iterate from there.

Pathology 6: collective bandwidth degrades over time during long runs

Cause: thermal throttling, NIC firmware issues, or connection-tracking buildup.

Fix: monitor IB error counters (ibcheckerrors), reset periodically, replace flaky NICs.

NCCL internals: how a collective actually executes

To debug NCCL effectively, it helps to know what's happening under the hood.

The communicator

NCCL operations happen within a communicator — a group of GPUs that can collectively communicate.

import torch.distributed as dist
dist.init_process_group("nccl")  # creates the default communicator

The communicator stores:

Rank IDs and connectivity graph.
Pre-computed routing tables for each algorithm.
Pre-allocated buffers for staging data.

Creating a communicator is expensive (hundreds of milliseconds). NCCL caches them across collectives.

A collective's lifecycle

Step-by-step for an all-reduce:

Caller invokes: dist.all_reduce(tensor) from Python.
NCCL plans: based on tensor size and topology, picks algorithm (Ring/Tree/NVLS) and protocol (LL/Simple).
NCCL launches kernels: CUDA kernels are queued on a CUDA stream.
Data movement: GPUs exchange data via NVLink, IB, or RoCE depending on path.
Kernels reduce: on each rank, partial sums are accumulated.
Kernels finalize: each rank now has the full reduced result.
CUDA stream syncs: the calling code blocks (or doesn't, if async) until the result is ready.

The Python call returns immediately after step 3. The actual data movement and reduction happen on GPU asynchronously. This is why NCCL operations can overlap with compute when used correctly.

Algorithm selection internals

NCCL selects algorithm based on:

if message_size < 4KB and ranks <= 8: use Tree+LL
elif message_size < 64KB: use Tree+Simple
elif NVLS supported and message_size < 8MB: use NVLS
else: use Ring+Simple

(Specific thresholds vary by NCCL version and topology.)

You can override:

export NCCL_ALGO=Ring     # always use Ring
export NCCL_ALGO=Tree     # always use Tree
export NCCL_ALGO=NVLS     # always use NVLS (errors if unsupported)

Override only if you have specific knowledge that defaults are wrong for your workload. Defaults are usually right.

How NCCL discovers topology

At init:

Detect all GPUs visible to each rank.
Probe NVLink connectivity between local GPUs.
Probe IB/RoCE connectivity between hosts.
Build a connectivity graph.
Choose default algorithm/protocol per (collective, message_size, rank_count).

This discovery takes several seconds. Verbose logging:

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=GRAPH

Shows the discovered topology. If it doesn't match your hardware, debug.

Customizing topology

If NCCL mis-detects (rare but possible), provide a topology file:

export NCCL_TOPO_FILE=/path/to/topology.xml

The XML format is documented in NCCL docs. Most users never need this.

NCCL with PyTorch DDP

PyTorch's DistributedDataParallel uses NCCL under the hood. Specifics:

Gradient bucketing

Instead of one all-reduce per parameter, DDP groups parameters into buckets and all-reduces them together:

ddp_model = DDP(model, bucket_cap_mb=25)  # 25MB per bucket

Larger buckets:

Better all-reduce efficiency (each NCCL call has fixed overhead).
Worse overlap with compute (need full bucket before reducing).

Smaller buckets:

More NCCL overhead.
Better overlap.

Sweet spot for most workloads: 25-50MB. For very large models on slow networks: 100MB+ to amortize.

Comm hooks

DDP supports custom communication hooks for special cases:

from torch.distributed.algorithms.ddp_comm_hooks import default_hooks

# Default: standard FP32 all-reduce
ddp_model.register_comm_hook(state, default_hooks.allreduce_hook)

# Compressed: 1-bit quantization
ddp_model.register_comm_hook(state, default_hooks.fp16_compress_hook)

For training across slow networks, compression hooks can help.

Find unused parameters

DDP(model, find_unused_parameters=True)

When some parameters don't see gradients in a step (common in MoE, conditional networks), enable this. Costs ~5% overhead but prevents hangs.

Async error handling

DDP(model, broadcast_buffers=False, gradient_as_bucket_view=True)

Modern flags that improve DDP's behavior under failures. Generally safe to enable.

NCCL with FSDP

FSDP (see PyTorch FSDP paper arXiv:2304.11277, and our companion distributed training guide) uses NCCL for both gradient sharding and parameter gathering.

All-gather pattern

For each layer, FSDP issues:

All-gather: collect this layer's full parameters from sharded ranks.
Forward/backward computation.
Reduce-scatter: scatter the gradient shards back.

These patterns are different from DDP's gradient all-reduce. NCCL handles both well, but the per-step collective count is higher.

Overlapping with compute

FSDP-2 added explicit overlap of all-gather with the next layer's compute. The pattern:

Layer N: gather params → forward → reduce gradients
                              ↓ (overlap)
Layer N+1: gather params (in flight)

When this works, FSDP throughput approaches DDP. When NCCL or PyTorch fails to overlap (some bug or environment issue), FSDP can be 2-3× slower than DDP for the same model.

Verify overlap with profiling. If you see large gaps between layers in the timeline, overlap isn't working.

NCCL on AMD: RCCL

AMD's equivalent is RCCL (ROCm Communication Collectives Library). API-compatible with NCCL — code that uses import torch.distributed as dist works on both.

Differences:

RCCL uses InfiniBand or RDMA-over-Ethernet (similar to NCCL).
AMD GPUs use Infinity Fabric for intra-node interconnect (analogous to NVLink).
Performance is generally competitive with NCCL on equivalent hardware.

For mixed clusters (uncommon but possible): NCCL and RCCL don't interoperate. Don't try to mix.

The bottom line

The named problem is the bandwidth bottleneck of synchronous SGD: gradient all-reduce sits on the critical path of every training step, and naive implementations make scaling N GPUs cost N times more communication. The solution is topology-aware collective communication — pick a ring, tree, or in-switch reduction that keeps per-link bytes roughly constant in N, and let the library match the transport (NVLink, NVSwitch, IB, RoCE, SHARP) to the message size. The single biggest lever is letting NCCL see your topology correctly — once it does, defaults are usually within a few percent of optimal.

What to do if you take only this away:

Run nccl-tests (all_reduce_perf -b 8 -e 8G -f 2 -g 8) on every cluster you touch and record the busbw curve. Anything below ~80% of link bandwidth at large sizes is a bug.
Set NCCL_DEBUG=INFO once at startup to confirm NCCL chose the transport you expect — NVLink intra-node, IB/RoCE inter-node, never PCIe or TCP unless intended.
On 1k+ GPU InfiniBand clusters, enable SHARP (NCCL_COLLNET_ENABLE=1) — it halves bytes on the wire for large all-reduces.
Pin versions: NCCL on every rank must match exactly, and the PyTorch wheel's bundled NCCL is what actually loads unless you override LD_LIBRARY_PATH.
Don't tune env vars blindly. Change one variable, re-run nccl-tests, keep what helps.

Next, read distributed LLM training for how DP, TP, PP, and FSDP each exercise these collectives differently, and NVLink and rack-scale topology for the hardware fabric NCCL is actually driving.

FAQ

Q: Should I always use NVLink for TP?

Yes. TP across InfiniBand is essentially never worth it. Stay within NVLink (single node or GB200 NVL72).

Q: How do I know if NCCL is using NVLink?

nvidia-smi topo -m shows topology. NV# = NVLink. PHB = PCIe. With NCCL_DEBUG=INFO, the log shows which path is chosen.

Q: Should I tune NCCL or trust defaults?

Trust defaults until you have evidence they're wrong. NCCL is well-tuned for common topologies.

Q: What's the difference between NCCL_ALGO and NCCL_PROTO?

ALGO is the high-level pattern (Ring, Tree, NVLS). PROTO is the wire-level protocol (LL, Simple). They combine.

Q: How does NCCL handle GPU failures during a collective?

Badly. NCCL has no native fault tolerance — a failed GPU hangs the collective. Modern frameworks (PyTorch's c10d with NCCL_TIMEOUT) provide an outer layer of timeout + restart.

Q: NCCL on AMD?

AMD's RCCL is API-compatible. Most code that uses NCCL works on RCCL with minimal changes. Performance is competitive but ecosystem maturity (debugging tools, documentation) lags.

Q: NCCL on Apple Silicon?

Not applicable. Apple Silicon uses MPS or Metal directly. No NCCL.

Q: How do I debug "NCCL hangs in init"?

Often network-related. Check IB/RoCE status, firewall, MPI launcher config. NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT shows where init is stalling.

Q: What if I have heterogeneous GPUs?

NCCL works fine across different GPUs (e.g., H100 + H200 in same job). Performance is dictated by the slowest GPU.

Q: Is NCCL stable?

Yes, very. NCCL is one of the most-tested pieces of NVIDIA's software stack. Bugs are rare; misconfiguration is common.

Q: How do I update NCCL?

NCCL ships in CUDA toolkit and PyTorch wheels. To upgrade: update PyTorch (it bundles a NCCL version). For finer control: install standalone NCCL and set LD_LIBRARY_PATH.

Q: NCCL across cloud regions?

Don't. Latency is too high. NCCL assumes datacenter-scale fabrics (sub-millisecond RTT). Cross-region collectives are not supported in any meaningful sense.

Q: How do I choose between NCCL and other collectives libraries?

For NVIDIA GPUs: NCCL. No real alternative.

For AMD GPUs: RCCL (NCCL-compatible API).

For non-GPU distributed: MPI, Gloo, others. Different design goals.

NCCL/RCCL dominate for GPU-based AI workloads.

Q: Should I use NCCL or MPI?

If you have GPUs and you're doing AI training, NCCL — full stop. MPI's collectives are not GPU-aware; even with CUDA-aware MPI builds, the implementation usually copies through host memory and loses 2-5× bandwidth versus NCCL's GPUDirect path. Use MPI as the launcher (mpirun, srun --mpi=pmix) if your cluster requires it, but let NCCL do the actual collectives.

The only places MPI's collectives still win in 2026 are pure-CPU jobs, sparse classical-HPC codes with occasional reductions, and frameworks that were architected pre-NCCL and haven't migrated.

Q: Does RCCL match NCCL performance?

On equivalent hardware and a well-tuned cluster, yes — within 5-10%. An 8× MI300X NVSwitch-equivalent box achieves ~600 GB/s AllReduce, comparable to ~750 GB/s on 8× H100 once you normalize for the lower xGMI bandwidth.

The gap that's still real in 2026 is operational: RCCL's debugging story (env vars, error messages, profilers) lags NCCL. Expect a 1-2 week tuning curve when you move to AMD that you wouldn't face on NVIDIA. SHARP-equivalent in-network reductions on AMD's Pollara network are also less mature than NVIDIA's Quantum + SHARP combination.

Q: When is Gloo sufficient?

Three cases. First, CPU-only distributed jobs — Gloo is the only PyTorch backend that works without CUDA. Second, development and debugging — Gloo runs over plain TCP, so you can prototype DDP on a laptop or non-RDMA dev cluster before deploying to the real fabric. Third, mixed CPU+GPU parameter servers, which are rare in 2026 but still exist for some recommendation-system workloads.

For production GPU training, Gloo is never the right answer — its GPU path is ~10× slower than NCCL.

Q: What's SHARP and when does it matter?

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) is NVIDIA's in-network reduction technology for InfiniBand Quantum switches. The switch fabric itself performs the AllReduce sum on the data passing through, so each GPU sends its tensor once instead of 2·(N-1)/N times.

When it matters: 256+ GPU clusters on Quantum / Quantum-2 / Quantum-3 IB. Below ~64 GPUs, the benefit is marginal because Ring is already fast enough. Above 1k GPUs, SHARP is essentially mandatory — it's the difference between a usable cluster and one where DP AllReduce dominates the step time.

Enable with NCCL_COLLNET_ENABLE=1 plus the SHARP daemon running on the fabric. Verify with NCCL_DEBUG=INFO — the log will show CollNet in the chosen algorithm.

Q: Does PyTorch c10d let me swap NCCL for RCCL or Gloo without code changes?

Almost. You change one string: init_process_group(backend="nccl") → "gloo" or "mpi". On ROCm builds, "nccl" already means RCCL. The collective API (all_reduce, all_gather_into_tensor, etc.) is identical across backends.

The catch is performance characteristics: a script tuned for NCCL's overlap behavior may bottleneck differently on Gloo or MPI. And not every collective is implemented in every backend — all_to_all is NCCL/MPI only; Gloo doesn't support it.

Q: How do JAX/XLA collectives compare to NCCL?

On TPUs, XLA's collectives run on Google's ICI torus and bypass NCCL entirely. The primitives are conceptually the same (psum, all_gather, all_to_all) but the lowering, layout, and scheduling happen in the XLA compiler, not at runtime. This is part of why TPUs feel different to program: you describe sharding declaratively (pjit, shard_map) and the compiler picks the collectives, instead of calling nccl.all_reduce imperatively.

On GPU backends, JAX still uses NCCL under the hood — XLA emits ncclAllReduce calls in the compiled HLO.

Q: How does NCCL handle ECN (Explicit Congestion Notification)?

NCCL respects ECN signals from the network. When ECN-marked packets indicate congestion, NCCL's flow control adjusts.

For RoCE deployments, ECN configuration matters. Properly configured ECN reduces tail latency.

Q: What's NCCL_PROTO=LL128?

A protocol variant optimized for NVLink. Provides lower latency than Simple for specific NVLink topologies.

NCCL auto-selects when appropriate; rarely needs manual override.

Q: Are there NCCL-style libraries from non-NVIDIA vendors?

AMD RCCL. Intel oneCCL. Each tied to vendor's GPU/CPU products.

For multi-vendor clusters: complexity. Most teams stick with one vendor.

Q: What's the role of NCCL in inference?

For multi-GPU inference (TP), NCCL handles per-layer activation all-reduces.

Inference workloads are typically lighter than training (fewer collectives, smaller messages). NCCL performance is rarely the bottleneck for inference.

Q: How does NCCL benefit from CUDA streams?

NCCL operations queue on CUDA streams. Multiple streams can have NCCL ops in flight simultaneously.

This allows compute-collective overlap (run compute on stream A while NCCL runs on stream B).

Production training relies on this for performance.

Q: What about NCCL with PyTorch's autograd?

PyTorch's DDP/FSDP handle NCCL integration with autograd. Backward pass triggers all-reduces; the framework batches them efficiently.

You don't typically interact with NCCL directly when using DDP/FSDP.

Q: How is NCCL different from Horovod?

Horovod was an early library for distributed training, predating PyTorch DDP. Used MPI + NCCL underneath.

In 2026, Horovod is largely deprecated for PyTorch users. PyTorch's native distributed training (DDP/FSDP) is the standard.

Q: What's the maximum cluster size NCCL handles?

In production: 10,000+ GPUs work fine. Frontier training has used NCCL on 16,000+ GPU clusters.

Theoretical limits are higher; practical limits are network-bound.

Q: How do I test NCCL with mixed precision?

NCCL works with any precision: BF16, FP16, FP8, FP32. The collective operates on the data as-is.

For all-reduce on quantized tensors: NCCL is content but precision loss can accumulate. Most production all-reduces gradients in BF16.

Q: What's the role of NCCL in MoE training?

NCCL provides the all-to-all collective used by EP. Critical for MoE performance.

NCCL's all-to-all has been optimized in recent versions for MoE-style routing patterns.

Q: Should I use NCCL for inference auto-scaling coordination?

No. NCCL is for tight-coupling collectives, not loosely-coupled coordination.

For inference scaling, use HTTP load balancing, gRPC, or coordination services (etcd, Consul).

Q: How does NCCL evolve?

NVIDIA releases new versions every 2-3 months. Major improvements:

New algorithms for new topologies.
Better adaptive routing.
More robust error handling.
Performance optimizations.

Stay current. Old NCCL versions miss meaningful improvements.

Q: What's the right NCCL version for production in 2026?

NCCL 2.20+ for Hopper. NCCL 2.22+ for Blackwell.

Pin a known-good version; don't auto-upgrade. Test new versions in staging first.

NCCL configuration recipes

Quick-reference configurations for common scenarios.

Recipe: 8-GPU single node H100 inference

export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=300
unset NCCL_P2P_DISABLE

That's it. Defaults are right.

Recipe: 32-GPU multi-node H100 fine-tuning

export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=1800
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_IB_GID_INDEX=3
export NCCL_BUFFSIZE=8388608

Recipe: 1024-GPU pre-training H100

Same as above plus:

export NCCL_NSOCKS_PERTHREAD=4
export NCCL_TREE_THRESHOLD=0
export NCCL_MAX_NRINGS=8

Recipe: GB200 NVL72 frontier training

export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=600
export NCCL_NVLS_ENABLE=1

Recipe: AWS EFA RoCE

export FI_PROVIDER=efa
export NCCL_PROTO=Simple,LL128
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_DISABLE=1

Recipe: debugging hangs

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log
export NCCL_TIMEOUT=300

Run with NCCL_TIMEOUT short; let it fail fast and inspect logs.

Recipe: maximum debug verbosity

export NCCL_DEBUG=TRACE
export NCCL_DEBUG_SUBSYS=ALL

Massive logs. Only for active investigation.

Common NCCL gotchas in production

Things that catch teams unawares.

CUDA out-of-memory during NCCL init

Symptom: torch.distributed.init_process_group fails with OOM.

Cause: insufficient GPU memory after weights loaded.

Fix: leave 2-4 GB free for NCCL buffers.

NCCL hangs after a node restart

Symptom: training hangs when one node was preempted/restarted.

Cause: stale state in NCCL communicator.

Fix: torch.distributed.destroy_process_group() and re-init.

Different NCCL versions on different nodes

Symptom: subtle slowdowns or hangs.

Cause: heterogeneous container images.

Fix: bake NCCL version into base image; verify version per-node.

Network MTU misconfiguration

Symptom: NCCL all-reduce slow but other tests OK.

Cause: MTU mismatch between NIC and switch.

Fix: match MTU end-to-end. 9000 (jumbo frames) is standard.

IB driver crashes

Symptom: NCCL errors in mid-training.

Cause: IB driver instability, often vendor-specific bugs.

Fix: update OFED. Check vendor errata.

Topology mis-detection

Symptom: wrong path chosen by NCCL.

Cause: NCCL's auto-detection has limits in unusual configs.

Fix: provide NCCL_TOPO_FILE explicitly.

CPU governor affecting performance

Symptom: NCCL latency varies with load.

Cause: Linux CPU frequency scaling.

Fix: set CPU governor to "performance" mode.

DPDK conflicts

Symptom: RoCE doesn't work after enabling DPDK.

Cause: DPDK and RoCE compete for NIC resources.

Fix: pick one. For NCCL: use kernel networking, not DPDK.

These all save time when you've seen them before.

Real-world cluster sizing

How frontier labs size their network.

For 256-GPU cluster (32 nodes)

8 NICs per node × 32 nodes = 256 NICs.
8 leaf switches (32 × 400 Gb/s ports each, full bisection per rail).
1 spine switch.
~1.5M total network cost.

For 1024-GPU cluster (128 nodes)

1024 NICs.
32 leaf switches.
8 spine switches.
~$3-4M network cost.

For 16,000-GPU cluster (2000 nodes)

16,000 NICs.
256 leaf switches (rail-optimized).
32 core switches.
$30-50M network cost.

Plus: optical cables (significant cost), power, cooling, configuration.

Network is 5-15% of total cluster cost. Not trivial.

NCCL collective performance benchmarks

Real-world numbers for sizing expectations.

All-reduce on different topologies

For a 1 GB tensor (typical FP32 gradient slice):

Topology	Algorithm	Achieved bandwidth	Latency
8× H100 NVLink (single node)	NVLS	1.2 TB/s	1ms
8× H100 NVLink (single node)	Ring	700 GB/s	1.5ms
8× H100 + 8× H100 (2 nodes IB NDR)	Hierarchical	80 GB/s	12ms
16× H100 (2 nodes RoCE)	Hierarchical	60 GB/s	18ms
1024× H100 (rail-optimized)	Tree+Ring	50 GB/s	25ms
8× B200 NVLink (single node)	NVLS	2.4 TB/s	0.5ms
72× B200 (GB200 NVL72)	NVLS	1.8 TB/s	1ms

NVLink within a node is enormously faster than across nodes. This is the structural reason for the within-node TP, across-node DP pattern.

All-to-all (MoE pattern)

For 1 GB total all-to-all data:

Topology	Achieved	Notes
8× H100 NVLink	800 GB/s	Effectively bisection-limited
8 nodes via IB NDR	30 GB/s	Bisection-bandwidth bound
GB200 NVL72	1.4 TB/s	Single-rack NVLink fabric

MoE training relies heavily on all-to-all. GB200 NVL72's rack-scale NVLink helps significantly.

Send/recv (PP pattern)

Pipeline parallelism uses point-to-point send/recv. Latency-bound for small messages.

Topology	Latency	Bandwidth
NVLink	1-2 µs	700 GB/s
IB NDR	4-6 µs	50 GB/s
RoCE	8-12 µs	50 GB/s
Cross-rack	10-20 µs	bandwidth-degraded

PP across racks is feasible but each pipeline stage adds latency.

NCCL on AWS EFA: complete guide

AWS EFA (Elastic Fabric Adapter) is AWS's RDMA-over-Ethernet implementation. NCCL works on it but with specifics.

Setup

For p5.48xlarge (8× H100):

# Install AWS EFA software
sudo apt install -y libfabric-bin libfabric-dev

# NCCL config
export FI_PROVIDER=efa
export NCCL_PROTO=Simple,LL128
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_DISABLE=1  # use EFA, not generic IB
export NCCL_DEBUG=WARN

EFA performance

Compared to dedicated NDR IB:

Latency: ~2× higher (EFA adds protocol overhead).
Bandwidth: 70-80% of theoretical NDR.
Tail latency: more variable than dedicated IB.

For workloads where this matters: dedicated cloud (CoreWeave) or on-prem.

For workloads where it's good enough: AWS EFA is mature and reliable.

Common EFA issues

"FI_EFA: failed to initialize": missing EFA driver or library. Check installation.

Slow all-reduce despite proper config: instance placement may put workers on different physical hosts. Use placement groups.

Random NCCL hangs on EFA: rare driver bugs. Update to latest AWS EC2 AMI.

When to choose EFA vs dedicated

AWS EFA: simpler ops, multi-region, AWS-integrated. Dedicated IB: better performance, lower latency, lower cost at scale.

For most AWS-native workloads: EFA. For frontier-scale dedicated: consider alternatives.

NCCL with InfiniBand: complete guide

For dedicated AI clusters with NVIDIA Quantum IB.

Setup essentials

Verify IB hardware:

lspci | grep -i infiniband
ibstat
ibhosts

Should show all HCAs and other hosts in the fabric.

NCCL config for IB

export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_IB_GID_INDEX=3
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_PCI_RELAXED_ORDERING=1

The 8 HCAs in NCCL_IB_HCA correspond to 8 IB adapters per node (rail-optimized topology).

GID index selection

GID (Global ID) determines which IB transport NCCL uses. Common values:

0: RoCE v1 (link-local).
1: IPv6.
2: RoCE v2 with RDMA over Ethernet.
3: RoCE v2 with IB transport (typical for InfiniBand fabrics).

For pure IB fabrics: GID 3.

For RoCE: GID 1 or 2.

If you're using the wrong GID, NCCL may work but at 50% expected bandwidth. Test with nccl-tests.

Verifying IB performance

ibv_rc_pingpong -s 1G  # IB ping-pong test

Round-trip latency should be ~1-2µs. Bandwidth at saturation should match link rate.

If RDMA isn't working: NCCL falls back to socket transport. Many times slower.

Subnet manager (OpenSM or UFM)

The subnet manager runs the IB fabric. Without it, IB doesn't work.

Most clusters use OpenSM (open-source) or NVIDIA UFM (commercial, more features).

Monitor SM health: failures cascade.

IB diagnostics tools

ibstat          # link status per HCA
ibhosts         # discover topology
ibcheckerrors   # error counters
ibtracert       # trace path between two nodes
ibping          # ping over IB

For production clusters: run these periodically. Catch issues before they manifest as slowdowns.

NCCL with RoCE: complete guide

RoCE works but requires careful configuration.

Setup essentials

For RoCE v2 over Ethernet:

# Identify RoCE-capable NICs
show_gids  # shows GID assignments

# Configure NCCL
export NCCL_IB_GID_INDEX=3  # for RoCE v2
export NCCL_IB_DISABLE=0

Lossless Ethernet requirements

RoCE assumes lossless network. Configure on switches:

Priority Flow Control (PFC): pause traffic on congestion. Configure for the priority class NCCL uses.
ECN (Explicit Congestion Notification): mark packets during congestion. NCCL responds by slowing down.

Without these, RoCE drops packets and NCCL retries → very slow.

Common RoCE issues

Half-bandwidth performance: typically wrong GID or PFC not configured. Run nccl-tests; if achieving <50% of theoretical, troubleshoot.

Random RoCE failures: switch firmware bugs or buffer overflow. Update firmware, verify switch buffer.

MTU mismatch: causes fragmentation. Use jumbo frames (9000 MTU) end-to-end.

Verifying RoCE performance

Same as IB: nccl-tests for end-to-end. ib_send_bw for low-level verification.

ib_send_bw -d mlx5_0 -F  # bandwidth test

Should achieve close to wire speed. If not, network configuration issue.

When RoCE works well

For modern Ethernet fabrics (Arista, Cisco, NVIDIA Spectrum):

Single-vendor stack with proper PFC/ECN.
< 10,000 GPU clusters.
Mid-tier deployments where IB cost premium isn't justified.

For frontier-scale: IB is more battle-tested.

Multi-node debugging cheat sheet

When a multi-node training run is slow.

Step 1: nccl-tests baseline

mpirun -np N -hostfile hosts.txt ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Compare achieved busbw to expected (per topology).

If 50%+ of expected: cluster is healthy. Issue is in your training code. If <50%: network issue. Continue diagnostics.

Step 2: identify slowest rank

# In your training code
import time
start = time.time()
torch.cuda.synchronize()
work = dist.all_reduce(tensor, async_op=True)
work.wait()
end = time.time()

print(f"Rank {rank}: all-reduce took {(end-start)*1000:.2f}ms")

If one rank is slower than others: investigate that rank's hardware/network.

Step 3: NCCL_DEBUG=INFO trace

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
python train.py 2>&1 | grep -E "INFO|WARN|ERROR" > nccl.log

Look for:

Which path NCCL chose (NVLink vs IB).
Algorithm selection (Ring vs Tree vs NVLS).
Any warnings or errors.

Step 4: per-collective timing

For a fully-instrumented training step, log per-layer collective time. If one layer is unusually slow, investigate.

Step 5: check for stragglers

If you have monitoring, check per-rank step time. Consistent stragglers = hardware issue.

Step 6: profile with Nsight

Capture a few steps with NVIDIA Nsight Systems. Visualize per-rank timeline.

This usually reveals where time is being spent.

Step 7: when stuck

If you've gone through all of the above and still don't know:

Check NCCL changelog for known issues with your version.
Review IB switch firmware for known bugs.
Engage NVIDIA enterprise support if available.
Try a different NCCL version.

Most issues are diagnosable with the steps above. Persistent issues are rare and usually firmware-level.

NCCL operational best practices

For teams operating production training infrastructure.

Version pinning

Pin NCCL version per cluster generation. Don't auto-update.

Test new versions in staging:

Run nccl-tests to verify performance hasn't regressed.
Run a small training job to verify functional correctness.
Roll out to production only after staging validation.

Container hygiene

If using containers:

Bake NCCL into base image (don't install at runtime).
Use the same image across all nodes in a cluster.
Verify image hash matches expected.

Mismatched images cause subtle bugs.

Logging strategy

Production: NCCL_DEBUG=WARN. Catches errors but not noisy.

Staging: NCCL_DEBUG=INFO. More info for testing.

Debugging: NCCL_DEBUG=TRACE. Max verbose, only when investigating.

Always log to file (NCCL_DEBUG_FILE) with hostname/PID in filename.

Timeout configuration

NCCL_TIMEOUT bounds collective hangs. Set conservatively:

Production training: 1800 seconds (30 min). Bounds without false positives.
Staging: 300 seconds. Faster failure detection.
Inference: 300 seconds. Inference is rarely hung that long without other issues.

Health checks

Before starting any large training job:

# Verify NCCL works on every node pair
mpirun -np N -hostfile hosts.txt \
  ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8 -c 1

Run as part of cluster startup procedure. Catches issues before training starts.

Monitoring metrics

Track at the framework level (PyTorch, etc.):

Per-step NCCL time.
Per-rank step duration variance.
Collective failure count.

Alert on:

NCCL time > 30% of step time.
Per-rank variance > 20%.
Any collective failure.

Pre-flight checks

Before deploying to production:

IB/RoCE connectivity verified per host.
NCCL version consistency verified.
nccl-tests baseline established.
Subnet manager (or RoCE PFC) verified.

A failed pre-flight check blocks deployment.

Rollback procedures

If a NCCL or driver update causes regression:

Have a known-good version pinned and ready to revert.
Maintain rollback playbook with specific commands.
Test rollback procedure quarterly.

Cluster issues are inevitable. Recovery time matters more than prevention.

On-call playbook

For on-call engineers responding to NCCL alerts:

Check overall cluster status.
Look for stragglers (single-node issues).
Run quick nccl-tests to verify cluster.
Check switch logs for fabric issues.
Escalate to vendor if needed.

Document the playbook. Train new on-call people.

Capacity headroom

Don't run cluster at 100% utilization. Reserve 5-10% for:

Recovery from straggler nodes.
Unexpected workload variability.
Maintenance and upgrades.

Cluster always at 95%+ utilization invariably has tail-latency issues.

NCCL implementation differences across versions

What changes when you upgrade.

NCCL 2.18 → 2.20 (Hopper era)

Better tree algorithm for medium messages.
CUDA Graph capture support.
Performance improvements on H100 NVSwitch.

Upgrade benefits: 10-20% on H100 cluster workloads.

NCCL 2.20 → 2.22 (Blackwell era)

B200 / GB200 support.
NVLS optimizations for rack-scale fabrics.
Improved adaptive routing.

For Blackwell deployments: required. For Hopper: incremental improvement.

Best version for your cluster

Hardware	Recommended NCCL
A100 cluster	2.18 (stable)
H100/H200 cluster	2.20+ (full Hopper support)
Mixed Hopper-Blackwell	2.22+ (Blackwell support)
GB200 NVL72	2.22+

Check NCCL release notes for cluster-specific issues before upgrading.

Compatibility considerations

NCCL versions need to match across all ranks. Mixed versions = potential issues.

PyTorch bundles a specific NCCL version. Override via LD_LIBRARY_PATH if you need different.

For long-running training: pin NCCL version in container; verify pre-flight.

Beyond NCCL: future of GPU communication

Where this is going.

MPI deprecation

For AI workloads, MPI was the original collective library (HPC heritage). NCCL has largely displaced it.

Some HPC sites still use MPI for compatibility. New AI deployments: NCCL.

Hardware-accelerated collectives

NVSwitch SHARP performs reductions in network. NVLS exposes this.

Future: more aggressive in-network compute. Some operations could entirely move to network.

CPU-GPU coherence

H100/B200 have NVLink-C2C for tight CPU-GPU coordination. Reduces some host-device transfers.

Future: more coherent memory across CPU and GPU. May change data movement patterns.

Async by default

CUDA Graphs and async operations make collectives implicit. Less explicit user control, more compiler optimization.

By 2027-2028: most collective decisions auto-optimized.

Cross-vendor

IBM, Intel, AMD all have collective libraries. Today incompatible with NCCL.

Possible future: standardized collective API across vendors. Slow progress.

Optical interconnects

Co-packaged optics may change what's possible. Higher bandwidth, longer reach.

May enable larger TP groups across multiple racks. Currently TP=72 in GB200 NVL72; could become TP=200+.

This area moves fast. NCCL keeps up.

Real-world tuning playbooks

Concrete configurations for common scenarios.

Playbook 1: 8-GPU DGX H100 inference

Single-node, NVLink-only, no inter-node concerns.

# Defaults are fine. Just enable NVLink P2P:
unset NCCL_P2P_DISABLE  # ensure not disabled
export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=300

# Run vLLM
vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 8

Expected: TP=8 all-reduce ~1ms latency, 700 GB/s sustained busbw.

Playbook 2: 64-GPU multi-node training (8 nodes × 8 H100)

Training a 70B model with TP=8 within node, DP=8 across nodes via InfiniBand.

# Each node:
export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=1800
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_IB_GID_INDEX=3
export NCCL_BUFFSIZE=8388608  # 8MB
export NCCL_NSOCKS_PERTHREAD=4
export NCCL_SOCKET_IFNAME=ibp0  # or eth0, depends on routing

torchrun --nnodes=8 --nproc_per_node=8 \
  --rdzv_backend=c10d --rdzv_endpoint=$MASTER_HOST:29500 \
  train.py

Expected: TP=8 within node ~50ms per all-reduce. DP=8 across nodes ~200ms per all-reduce on NDR IB.

Playbook 3: GB200 NVL72 rack-scale

Rack-scale NVLink fabric. TP=72 within one fabric.

# All defaults work. NCCL auto-detects rack-scale fabric.
export NCCL_DEBUG=WARN
export NCCL_TIMEOUT=600

# Specific to GB200:
export NCCL_NVLS_ENABLE=1  # in-network reductions on NVSwitch

Expected: TP=72 all-reduce <100ms with NVLS.

Playbook 4: RoCE on AWS p5 instances

EFA-based RoCE on AWS.

export FI_PROVIDER=efa
export NCCL_PROTO=Simple,LL128
export NCCL_NET_GDR_LEVEL=5
export NCCL_IB_DISABLE=1   # use EFA, not generic IB

EFA performance is generally 70-80% of dedicated NDR IB. Some workloads see more gap; benchmark.

Playbook 5: cross-rack training (>1024 GPUs)

Beyond a single rail-optimized cluster, hierarchical tuning.

export NCCL_TREE_THRESHOLD=0  # force Ring for large messages (better at scale)
export NCCL_MAX_NRINGS=8       # multiple parallel rings for bandwidth
export NCCL_BUFFSIZE=16777216  # 16MB

Frontier-scale training (16k+ GPUs): every parameter matters. NVIDIA provides specific tuning guides per cluster type.

Debugging mismatched NCCL versions

A subtle issue: different NCCL versions across ranks can produce silent slowdowns or hangs.

Detection

python -c "import torch; print(torch.cuda.nccl.version())"

Run on every node before starting a job. All should match.

Common version mismatch causes

Different container images on different nodes.
Bare-metal vs containerized mix.
Recent NCCL upgrades not propagated everywhere.
Different PyTorch versions bundle different NCCLs.

Fixes

Pin the same container image across all nodes.
Use LD_LIBRARY_PATH to point all ranks at one canonical NCCL install.
Verify with nccl-tests's version output before starting training.

NCCL and CUDA Graphs

CUDA Graphs capture sequences of CUDA ops and replay them with minimal overhead. NCCL collectives can be captured into graphs since NCCL 2.18.

For training, this can reduce per-step Python and CUDA dispatch overhead by 30-50%.

# In Megatron-LM with CUDA Graphs enabled:
mpu.set_use_cuda_graphs(True)

Caveats:

All collectives in a graph use the same NCCL communicator.
Dynamic shapes (varying batch sizes) defeat graph capture.
Some NCCL features (debug logging) are incompatible with graphs.

For large-scale training where Python overhead is a bottleneck, CUDA Graphs are a real win. Most modern frameworks integrate them. See our CUDA Graphs and torch.compile guide for the inference side.

Multi-rail and adaptive routing

Modern InfiniBand fabrics support adaptive routing: switches dynamically pick paths based on congestion. Improves all-reduce performance under load.

To enable:

export NCCL_IB_AR_THRESHOLD=8192

This makes NCCL allow adaptive routing for messages above 8KB. Typically a 5-15% improvement on congested fabrics.

Topology-specific:

NVIDIA Quantum InfiniBand (HDR/NDR): supports adaptive routing.
AWS EFA: limited.
RoCE: depends on switch firmware.

Test before relying on it. Some old fabrics misbehave with adaptive routing enabled.

NCCL on heterogeneous hardware

What if your cluster has mixed GPUs (H100 + H200, or H100 + A100)?

NCCL works across heterogeneous GPUs. But:

Performance is dictated by the slowest GPU in any TP group.
Memory differences mean per-GPU batch sizes can't differ within a TP group.
Some collective patterns (NVLS) require all GPUs in the group to support it.

For mixed clusters: keep TP groups homogeneous (e.g., all H100s in one TP group, all H200s in another). DP across the mix is fine.

NCCL profiling deep dive

When debugging slow NCCL, profiling is essential.

NCCL's built-in tracing

export NCCL_DEBUG=TRACE
export NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log

Produces detailed per-collective traces. Massive logs; only use for debugging specific incidents.

NVIDIA Nsight Systems

Profile entire training step including NCCL.

nsys profile --trace=cuda,nvtx,osrt \
  --stats=true \
  --capture-range=cudaProfilerApi \
  python train.py

Open .qdrep in Nsight UI. NCCL collectives show as labeled spans. Look for:

Long collective time relative to compute.
Collectives serialized when they should overlap.
Idle GPU time around collectives.

Aggregating across ranks

For multi-node training, collect profiles from each rank:

nsys profile --capture-range=cudaProfilerApi \
  --output=trace-%p-rank-%q{RANK} \
  python train.py

Then merge with nsys stats --report=... to see per-collective metrics across the cluster.

Common patterns

Slowest rank dictates step time: classic straggler pattern. All ranks finish at the same time, but the slowest one was working harder. Investigate that node's hardware.

Communication > compute: NCCL collectives dominate step time. Either reduce TP/PP, or fix fabric performance.

Imbalanced collective time: same collective takes very different times on different runs. Network instability or congestion.

Compute-collective overlap broken: per-rank timeline shows compute pausing during collectives. Async overlap isn't working. Check the framework's overlap settings.

Glossary

all_reduce: collective that sums and broadcasts.
all_to_all: collective where every rank sends to every other.
busbw: bus bandwidth, the relevant metric in nccl-tests.
GDR: GPU Direct RDMA. Direct DMA between GPU and NIC.
HCA: Host Channel Adapter (IB term for NIC).
IB: InfiniBand. NVIDIA's preferred RDMA fabric.
LL / LL128 / Simple: NCCL transport protocols.
NCCL: NVIDIA Collective Communications Library.
NVLink: GPU-to-GPU interconnect within a node.
NVLS: NVLink SHARP. Hardware-accelerated reduction.
NVSwitch: NVLink fabric switch.
P2P: peer-to-peer (GPU-to-GPU direct).
PFC: Priority Flow Control. Lossless Ethernet feature.
rail: dedicated network path in rail-optimized topology.
RDMA: Remote Direct Memory Access. Bypass CPU on memory transfers.
RoCE: RDMA over Converged Ethernet.

References

NVIDIA, NCCL Documentation, https://docs.nvidia.com/deeplearning/nccl/.
NVIDIA, nccl-tests Repository, https://github.com/NVIDIA/nccl-tests.
NVIDIA, NCCL Tuning Guide, periodically updated technical brief.
AMD / ROCm, RCCL Repository, https://github.com/ROCm/rccl.
Intel, oneCCL Repository, https://github.com/oneapi-src/oneCCL.
MPI Forum, MPI Standard, https://www.mpi-forum.org/.
Meta / Facebook, Gloo Repository, https://github.com/facebookincubator/gloo.
NVIDIA, SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) Documentation, https://docs.nvidia.com/networking/display/sharpv301.
PyTorch, Distributed Communication Package (c10d), https://pytorch.org/docs/stable/distributed.html.
Sergeev & Del Balso, Horovod: Fast and Easy Distributed Deep Learning in TensorFlow, 2018, arXiv:1802.05799.
Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017, arXiv:1706.02677.
Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019, arXiv:1909.08053.
Patarasuk & Yuan, Bandwidth Optimal All-Reduce Algorithms for Clusters of Workstations, JPDC 2009.
Jiang et al., A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (BytePS), 2020, arXiv:2006.01987.
DeepSeek-AI, DeepSeek-V3 Technical Report (engineering on collectives at scale), 2024, arXiv:2412.19437.

NCCL deep dives by topology

How NCCL behaves under different physical topologies.

Single-node (8-GPU) baseline

Within a Hopper or Blackwell node:

8 GPUs share an NVSwitch fabric.
All-to-all NVLink bandwidth: 900 GB/s per GPU (Hopper), 1,800 GB/s (Blackwell).
NCCL uses ring or tree algorithm intra-node.
Latency: ~5-10μs for small ops.

For most workloads: this is the highest-bandwidth, lowest-latency configuration.

Two-node cluster

When you cross node boundaries:

Inter-node bandwidth = NIC bandwidth (typically 400 Gbps NDR or 800 Gbps XDR).
This is ~2 orders of magnitude less than NVLink.
NCCL detects topology and adjusts: intra-node first, then inter-node.

Performance impact: collective ops involving both nodes scale to inter-node bandwidth.

For training: can be acceptable if compute > comms. For inference (TP across nodes): usually problematic.

Mid-scale (16-64 nodes)

At this scale:

Network topology starts to matter.
Fat-tree / non-blocking fabric is preferred.
Rail-optimized helps but isn't essential yet.

NCCL's auto-tuning typically works well at this scale.

Large-scale (128-512 nodes)

Critical considerations:

Rail-optimized topology recommended.
NCCL_TOPO_FILE for explicit topology hints.
SHARP for in-network reductions.

Performance degradation without proper tuning: 30-50%.

Frontier-scale (1k+ nodes)

What changes:

SHARP becomes essential (factor 2-3x improvement).
Hierarchical algorithms reduce cross-rack traffic.
Network failures are routine — NCCL must handle.
Co-design between hardware, NCCL, and application.

Examples: Meta's Llama-3 cluster, xAI Colossus.

NCCL operational insights

Operational lessons from running NCCL at scale.

Restart frequency

In a 1k-GPU cluster, expect:

Daily: 1-3 GPU failures or transient errors.
Weekly: minor network anomalies.
Monthly: significant fabric event.

Build automation around this. Don't expect "uptime" in traditional sense.

NCCL state during failures

When a NIC drops:

Outstanding NCCL ops may hang indefinitely.
Recovery requires teardown of all NCCL communicators.
Application-level checkpoint/restart is the only reliable recovery.

Plan for: aggressive checkpointing (every 100-1000 steps).

Healthy NCCL fingerprint

A healthy cluster shows:

Consistent collective times across iterations (variance < 5%).
No warnings/errors in NCCL_DEBUG=WARN log.
nccl-tests results within 5% of theoretical.

Anomalies indicate issues to investigate.

Common operational mistakes

Skipping topology validation: assuming auto-tuning handles all cases.
Insufficient warm-up: first iterations can be slow; benchmark steady-state.
Mixing NCCL versions: ensure all nodes have same version.
Network buffer misconfiguration: kernel/sysctl parameters affect.
Ignoring environment: NCCL is sensitive to many env vars.
No baseline benchmarks: without baseline, regressions go undetected.

NCCL upgrade discipline

NCCL evolves rapidly. Each release:

Fixes bugs.
Adds optimizations.
Sometimes changes default behavior.

Upgrade strategy:

Test in dev cluster.
Run nccl-tests before/after.
Monitor first day of production with alerting.
Keep rollback plan ready.

Skipping upgrades: misses performance improvements. Upgrading too fast: risks regressions.

Sweet spot: stable releases, ~1 month after release.

NCCL and other communication libraries

How NCCL relates to alternatives.

MPI (Message Passing Interface)

The classic. Used by HPC for decades.

Strengths: portable, well-understood, many implementations. Weaknesses: not GPU-optimized; some implementations are slow on modern hardware.

NCCL vs MPI: NCCL faster for GPU collectives. MPI useful for CPU collectives or non-NVIDIA GPU clusters.

Gloo

Facebook's collective library. Used by PyTorch as fallback.

Strengths: simple, handles CPU and GPU. Weaknesses: slower than NCCL on NVIDIA GPUs.

Use case: when NCCL isn't available (e.g., AMD GPU clusters, mixed CPU/GPU).

RCCL

AMD's NCCL equivalent for ROCm.

Generally NCCL-compatible API. Performance similar within AMD ecosystem.

oneCCL

Intel's collective library for GPUs and CPUs.

Used in Intel Gaudi clusters and Intel GPU deployments.

Collective Communications Library evolution

Industry trend toward standardization:

UCC (Unified Collective Communication) abstracts multiple backends.
PyTorch supports multiple backends transparently.

Future: likely more standardization, less vendor-lock-in.

NCCL future directions

Where NCCL is heading.

Tighter SHARP integration

In-network reductions (SHARP) move computation into switches. Currently optional, becoming default for large clusters.

Future versions: deeper SHARP integration, more collective operations supported.

Multi-tenancy support

As GPU clusters become multi-tenant, NCCL needs better isolation:

QoS for collective traffic.
Tenant-aware scheduling.
Bandwidth fairness.

This is active work.

Heterogeneous clusters

NCCL traditionally assumes homogeneous GPUs. Real clusters increasingly have mixed Hopper + Blackwell, etc.

Future: better handling of heterogeneous configurations.

Open-source contributions

NCCL is open-source (NVIDIA-led). Community contributions are increasing.

This is healthy for ecosystem evolution.

Integration with newer fabrics

UEC (Ultra Ethernet Consortium) and other fabric standards:

NCCL needs to support these as they emerge.
Performance characteristics may differ from IB.

Active integration work.

NCCL FAQ extension

Real-world NCCL case studies

How NCCL behaves in production deployments.

Case study 1: Llama-3 405B training

Meta trained Llama-3 405B on 16k H100s.

NCCL details:

All-reduce dominant (data parallel + tensor parallel mix).
SHARP enabled for in-network reductions.
Custom NCCL_TOPO_FILE for rail-optimized fabric.

Lessons:

At this scale, ~25% of wall-clock time was spent on collectives.
NCCL tuning made the difference between 40% MFU and 50% MFU.
Operational maturity (auto-recovery, monitoring) was critical.

Case study 2: Mixture-of-Experts training

For MoE models like Mixtral or DeepSeek:

Expert parallelism uses all-to-all heavily.
NCCL all-to-all benefits from rail-optimized topology.
Capacity factors and load imbalance affect collective time.

Lessons:

Tune for all-to-all in addition to all-reduce.
Profile expert load distribution.
Consider hierarchical all-to-all for skewed loads.

Case study 3: Large-scale inference (TP=8 across 1k requests/sec)

When serving Llama-3 70B with TP=8:

Every token requires NCCL all-reduce.
Latency per all-reduce: 50-200μs depending on size.
This is added to per-token latency.

Lessons:

For latency-critical inference, prefer single-node TP.
CUDA Graphs help reduce overhead.
NCCL warmup at server startup.

Case study 4: Failure recovery in long training runs

Real story: a 30-day training run experienced 47 NCCL-related events.

Mitigation:

Checkpoint every 1000 steps (~30 min).
Auto-restart from last good checkpoint.
Health monitoring catches stragglers.

Result: 95% effective uptime on 1k-GPU cluster.

Case study 5: Heterogeneous cluster operation

A mixed Hopper + Blackwell cluster:

NCCL needed to handle different GPU generations.
Bandwidth differs between nodes.
Auto-tuning didn't always pick the best algorithm.

Mitigation: explicit topology file with bandwidth annotations.

Lesson: heterogeneous clusters require more tuning than homogeneous.

NCCL performance tuning playbook

A step-by-step playbook for tuning NCCL performance.

Step 1: Baseline measurement

Run nccl-tests with default config. Record:

All-reduce performance at various message sizes.
All-gather, reduce-scatter, all-to-all where applicable.
Variance across iterations.

This is your baseline.

Step 2: Theoretical analysis

Calculate theoretical bandwidth:

Algorithm bandwidth (algbw): expected for the algorithm.
Bus bandwidth (busbw): hardware-limited.

Compare measured vs theoretical. Gap indicates room for improvement.

Step 3: Algorithm selection

Try different algorithms:

NCCL_ALGO=Ring: standard, generally good for large messages.
NCCL_ALGO=Tree: better for small messages or when latency matters.
NCCL_ALGO=NVLS: when SHARP/NVLink-SHARP is available.

Re-run nccl-tests and compare.

Step 4: Protocol tuning

Try different protocols:

NCCL_PROTO=Simple: standard.
NCCL_PROTO=LL: low-latency for small messages.
NCCL_PROTO=LL128: optimized for NVLink.

Each has different tradeoffs.

Step 5: Buffer size tuning

NCCL_BUFFSIZE affects performance:

Default: 4MB.
Larger buffers: better for big messages.
Smaller buffers: better for many small messages.

Tune based on your workload's message size distribution.

Step 6: NIC optimization

For multi-node:

NCCL_IB_HCA: which HCA(s) to use.
NCCL_IB_GID_INDEX: GID for routing.
NCCL_IB_TC: traffic class for QoS.
NCCL_NET_GDR_LEVEL: GPU Direct RDMA usage.

Verify each NIC is being used.

Step 7: SHARP enablement

If hardware supports SHARP:

NCCL_COLLNET_ENABLE=1.
Set up SHARP daemons.
Validate via NCCL_DEBUG=INFO.

Can yield 2-3x speedup on large clusters.

Step 8: Application-level tuning

Beyond NCCL itself:

Batch communication operations.
Overlap compute and communication.
Use gradient bucketing.
Profile to identify bottlenecks.

These often have larger impact than NCCL tuning.

Step 9: Continuous monitoring

After tuning:

Track collective time per iteration.
Alert on regressions.
Re-run nccl-tests periodically.

Tuning isn't one-time — it's continuous.

Step 10: Document and share

Document your tuning. Share with your team.

NCCL knowledge is valuable; preserve it institutionally.

NCCL anti-patterns

What not to do.

Anti-pattern 1: Aggressive timeout tuning to mask hangs

Setting NCCL_TIMEOUT to a very small value to fail fast hides real issues.

Better: investigate why hangs occur, fix root cause.

Anti-pattern 2: Skipping nccl-tests baseline

Without a baseline, you can't tell if production performance is healthy or degraded.

Better: nccl-tests is mandatory for any new cluster.

Anti-pattern 3: Random env var copying

Copying NCCL env vars from a Stack Overflow answer without understanding can hurt.

Better: understand what each variable does. Test before/after.

Anti-pattern 4: Mixing different NCCL versions across nodes

Causes subtle issues. Hard to debug.

Better: ensure all nodes have same NCCL version. Pin in container images.

Anti-pattern 5: Ignoring NCCL warnings

NCCL warnings often indicate real issues. Don't suppress them.

Better: investigate warnings. Fix underlying issues.

Anti-pattern 6: Treating NCCL as opaque

NCCL is documented and open-source. Reading source is sometimes the fastest path to understanding.

Better: when stuck, read the NCCL source.

Anti-pattern 7: Not isolating NCCL traffic

NCCL traffic competing with other workloads degrades performance.

Better: separate NICs for NCCL, or QoS to prioritize.

Anti-pattern 8: Skipping warmup

First NCCL ops can be slow due to setup. Production benchmarks should be steady-state.

Better: always warmup before measuring.

Anti-pattern 9: Over-engineering for single-node

Most NCCL complexity is multi-node. Single-node deployments don't need it.

Better: keep config simple for single-node.

Anti-pattern 10: Not testing failover

Plan for failures. Test recovery procedures.

Better: regularly chaos-test recovery.

NCCL configuration recipes

Battle-tested configurations for common scenarios.

Recipe: Single-node 8x H100

# No special config needed. NCCL auto-detects.
export NCCL_DEBUG=WARN

Recipe: Multi-node 8x H100, InfiniBand

export NCCL_DEBUG=WARN
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
export NCCL_IB_GID_INDEX=3
export NCCL_NET_GDR_LEVEL=2
export NCCL_TOPO_FILE=/etc/nccl/topology.xml

Recipe: Multi-node 8x H100, RoCE

export NCCL_DEBUG=WARN
export NCCL_IB_HCA=mlx5_0,mlx5_1
export NCCL_IB_TC=106
export NCCL_IB_SL=3
export NCCL_NET_GDR_LEVEL=2

Recipe: Large cluster with SHARP

export NCCL_COLLNET_ENABLE=1
export NCCL_SHARP_ENABLE_NIC_USAGE=1
# ... plus standard IB config

Recipe: Inference (TP=8 single node)

export NCCL_DEBUG=WARN
export NCCL_NTHREADS=128
# CUDA Graphs handle most overhead

These recipes are starting points. Always tune for your specific hardware.

Algorithm bandwidth math: what each collective costs

Knowing what NCCL should achieve on paper is what separates "feels slow" from "actually slow." Every collective has a closed-form bandwidth ceiling derived from the algorithm and the topology — match against nccl-tests busbw to know whether you have a tuning problem or a physics problem.

AllReduce bandwidth ceilings

For Ring AllReduce of message M across N ranks on links of bandwidth B:

Bytes per link per rank: 2·(N-1)/N · M. For large N, that approaches 2·M.
Best-case wall-clock time: 2·(N-1)/N · M / B.
Reported busbw in nccl-tests is M / time scaled by 2·(N-1)/N — so a healthy Ring AllReduce reports busbw close to the per-link B.

For Tree AllReduce on small messages, the latency floor is 2·log2(N) · α + M/B, where α is the per-step RTT (1-2 µs on NVLink, 4-10 µs on IB). Below ~64 KB this beats Ring because the α·log N term wins over Ring's α·N.

For CollNet (SHARP), bytes-on-the-wire drop to M (each rank sends once into the switch, receives the reduced result once). Theoretical 2× speedup over Ring on bandwidth-bound regimes.

A 1 MB AllReduce on 8× H100 NVSwitch, worked out

NVLink-4 per-GPU bidirectional bandwidth: 900 GB/s. Ring on 8 ranks moves 2·(7/8)·1 MB = 1.75 MB of traffic per rank, at ~700 GB/s sustained — completion time ≈ 2.5 µs. Add ~5 µs of kernel-launch and synchronization overhead, and nccl-tests will report ~7-8 µs end-to-end with busbw ~700 GB/s. If you see 200 GB/s instead, NCCL is on the PCIe path. If you see 1.2 TB/s, NVLS is engaged and you're getting in-switch reduction.

A 1 GB AllReduce on 64 nodes via 400 Gbps IB NDR

Per-port bandwidth: 50 GB/s. Hierarchical AllReduce reduces intra-node first (NVLink, fast), then 1 GB across 64 nodes via the inter-node ring. Inter-node bytes per rank: 2·(63/64)·1 GB ≈ 1.97 GB. At 50 GB/s per rail with 8 rails per node, effective per-direction bandwidth ≈ 350 GB/s. Completion ≈ 6 ms. SHARP cuts that to ~3 ms by halving bytes-on-the-wire.

When measured busbw lies

nccl-tests busbw is computed as (algorithm-specific factor) × M / time, so a wrong algorithm pick can show a misleading number. If NCCL_ALGO=Tree is forced for a 1 GB message, the algorithm factor is wrong for the actual data motion pattern and busbw can read artificially low. Always cross-check by also looking at algbw (algorithmic bandwidth) and by computing M/time by hand for the regime you care about.

When to override NCCL defaults

NCCL's auto-tuning is good. The cases where overriding pays off are specific and few.

Force Ring for very large messages on small clusters

On 4-8 ranks within an NVSwitch fabric, Ring achieves close to link bandwidth on >4 MB messages. NCCL may pick NVLS, which is faster on paper but sometimes loses on systems with old NVSwitch firmware. If NCCL_DEBUG=INFO shows NVLS and nccl-tests reports lower busbw than expected, try NCCL_ALGO=Ring and compare.

Force Tree for tiny gradient reductions in RLHF

PPO-style training has many small AllReduces (KL divergences, reward statistics) under 64 KB. NCCL usually picks Tree here, but on some configurations it switches to Ring too early. NCCL_TREE_THRESHOLD=131072 forces Tree up to 128 KB and can cut per-step latency by 5-10% for these workloads.

Disable NVLS when debugging numerical issues

NVLS performs reductions in NVSwitch silicon; the reduction order is different from Ring. If you're hunting determinism bugs, NCCL_NVLS_ENABLE=0 forces software reduction and gives you bit-identical results across runs at the same rank count. Re-enable in production.

Raise NCCL_BUFFSIZE for very-large-message workloads

For M > 256 MB (long-context activations in TP, very large optimizer states in ZeRO-3), bumping NCCL_BUFFSIZE from 4 MB to 16 MB or 32 MB increases pipelining and can add 10-20% throughput. The cost is per-rank GPU memory: NCCL_BUFFSIZE × num_channels × num_peers. At 16 MB and 8 channels on 8 ranks, that's ~1 GB of NCCL-reserved memory.

When not to override

Defaults already adapt to topology and message size. Overriding without nccl-tests evidence of a specific regression usually loses performance, not gains it. The most damaging override is NCCL_P2P_DISABLE=1 left over from an old debugging session — it routes everything through host memory and turns a 700 GB/s collective into a 30 GB/s collective.

NCCL vs UCX vs libfabric: the transport layer story

NCCL is the collective layer; the actual bytes ride on a transport layer. Which transport is in play changes performance characteristics and which env vars matter.

NCCL's built-in transports

NCCL has its own IB verbs implementation (net_ib), a socket transport (net_socket), and a plugin interface for vendor-specific networks. On any standard NVIDIA + Mellanox + IB cluster, NCCL talks IB verbs directly — no MPI, no UCX. This is the fastest path and what every frontier lab uses.

UCX as a backend

Unified Communication X is a portable transport library used by OpenMPI, MPICH, and increasingly Intel and AMD stacks. NCCL has historical UCX plugin support but it's deprecated for NVIDIA hardware — direct verbs is faster. UCX still matters when your cluster's network is exotic (Cray Slingshot, Atos BXI) and NCCL needs a portable backend to ride on.

libfabric and AWS EFA

AWS Elastic Fabric Adapter exposes a libfabric (FI_PROVIDER=efa) interface, not IB verbs. NCCL uses NVIDIA's AWS-OFI-NCCL plugin to talk libfabric. Performance is 70-85% of equivalent-bandwidth IB depending on instance type and placement group quality. Specifics for AWS EFA are covered above.

Transport selection priority

NCCL picks transports in this order: NVLink P2P (intra-node) → CUDA IPC (intra-node fallback) → IB verbs (inter-node, if present) → AWS-OFI plugin (if FI_PROVIDER set) → TCP sockets (last resort). The first viable option wins. NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=NET shows the chosen path per peer.

Transport comparison

Transport	Where it's used	Per-direction BW (typical)	Latency floor	Notes
NVLink + NVSwitch	Intra-node H100/B200	450-900 GB/s	1-2 µs	First choice; never disable
CUDA IPC	Intra-node PCIe-only	30-50 GB/s	3-5 µs	Fallback when NVLink missing
IB verbs (NDR 400G)	Inter-node, NVIDIA Quantum	45-50 GB/s	1.5-2 µs	Frontier default
IB verbs (XDR 800G)	Quantum-3 (2025+)	90-95 GB/s	1.5 µs	Newest, expensive
RoCE v2 + PFC	Inter-node, Ethernet	30-45 GB/s	3-5 µs	Cloud and mid-tier
AWS-OFI EFA	AWS p5/p5e instances	30-40 GB/s	8-15 µs	Cloud workhorse
Cornelis Networks OPA	Niche HPC	20-25 GB/s	1-2 µs	Rare in AI
TCP sockets	Anything else	1-10 GB/s	30-100 µs	Disaster path

Tuning NCCL for specific frameworks

The framework that calls NCCL shapes what tuning matters. Quick guidance per stack.

PyTorch DDP

DDP uses gradient bucketing; bucket size is the dominant knob, not NCCL settings. Set bucket_cap_mb=25 for most models, 50-100 for 100B+ models on slow networks. NCCL's NCCL_NTHREADS matters only at very small message sizes where Python overhead competes with collective overhead — leave at default. For an end-to-end DDP recipe see distributed LLM training.

PyTorch FSDP / FSDP2

FSDP issues AllGather (forward) and ReduceScatter (backward) per layer. The collective count is 2 × num_layers, often 100+ small-to-medium collectives per step. NCCL's per-launch overhead matters more than for DDP. Enable NCCL_BLOCKING_WAIT=0 (default) for async; set NCCL_BUFFSIZE=8388608 (8 MB) to give more pipelining room. For very large models, FSDP-2's explicit prefetch combined with NCCL_LAUNCH_MODE=GROUP reduces dispatch overhead by 20-30%.

Megatron-LM tensor parallelism

TP issues two AllReduces per transformer layer (attention output, MLP output) plus optional sequence-parallel ReduceScatter/AllGather. Within a node, NVLink + NVLS handles it; never extend TP across nodes — IB latency dominates. NCCL_TREE_THRESHOLD=0 (force Ring) can win on H100/B200 TP=8 because the messages are large (hidden_dim × seq_len × dtype_bytes, often 50-200 MB).

DeepSpeed ZeRO-3

ZeRO-3 has a similar AllGather/ReduceScatter pattern to FSDP but issues collectives at a coarser granularity (the entire optimizer state). Larger messages mean Ring dominates and NCCL_BUFFSIZE tuning matters more. For ZeRO-Infinity offload, NCCL contends with PCIe traffic to NVMe — pin NCCL to specific cores via NCCL_IGNORE_CPU_AFFINITY=0 and taskset.

vLLM tensor parallelism

vLLM issues one AllReduce per transformer layer in decode (small messages, latency-bound) and per-prefill (larger messages, bandwidth-bound). Decode latency dominates SLOs, so Tree algorithm matters: don't disable Tree; don't force Ring. CUDA Graph capture (vLLM enables by default since 0.4.0) amortizes NCCL launch overhead.

JAX / XLA on GPU

XLA compiles collectives into the HLO; the XLA flag --xla_gpu_enable_async_collectives=true enables overlap with compute. The compiled NCCL call ignores most runtime env vars at planning time, so set NCCL env vars before JAX imports — late changes won't take effect until the next process.

NCCL determinism and reproducibility

A frequent gotcha when chasing eval reproducibility.

Why NCCL is non-deterministic by default

Floating-point addition isn't associative. Different reduction orders → different results. NCCL's algorithm picker can pick different paths based on topology probes that complete in different orders run-to-run, leading to bit-different outputs.

Forcing deterministic NCCL

export NCCL_ALGO=Ring          # fix the algorithm
export NCCL_PROTO=Simple       # fix the protocol
export NCCL_NVLS_ENABLE=0      # disable in-switch reduction
export CUBLAS_WORKSPACE_CONFIG=:4096:8

Combined with framework-side determinism (torch.use_deterministic_algorithms(True)), this gets you bit-identical training across runs at the same rank count. Change the number of ranks and reductions still differ — the only fix is full FP64 reduction (impractically slow) or post-hoc Kahan summation.

Cost of deterministic NCCL

Forcing Ring on small messages loses 5-15%; disabling NVLS loses 30-40% on AllReduce-heavy workloads. Use only when reproducibility is mandatory (regulated evals, audit-grade training runs). For most production training, accept the noise; checkpoint frequently and compare loss curves at coarse granularity.

NCCL versus collective communication on TPUs

A common question from teams evaluating Google TPUs versus NVIDIA GPUs.

TPU ICI: a different design

Google's Inter-Chip Interconnect is a 3D torus (TPU v4/v5) or 2D mesh (v5e) directly between TPU dies, with no switch. Bandwidth per link is 100-450 GB/s depending on generation. There's no library equivalent to NCCL — XLA compiles psum, all_gather, etc. directly to torus moves.

Performance comparison at scale

On a 256-chip TPU v5p pod, AllReduce of 1 GB completes in ~~2 ms — competitive with 256× H100 over 400G IB (~~3 ms with SHARP). On 4096-chip pods, the torus topology means worst-case hops scale as O(N^(1/3)) instead of O(log N) for a fat-tree, which trades off badly at extreme scale. NVIDIA wins on flexibility (any topology, any vendor) and on raw compute density per chip; Google wins on simplicity (no separate fabric to tune) and on bisection cost-efficiency at sub-1000-chip scale.

When TPU collectives are the right answer

If you're already JAX-native and your model fits in a pod, ICI removes most of the tuning surface this guide describes. If you're PyTorch-native with custom CUDA kernels, the migration cost is rarely worth the collective-layer savings.

Advanced env vars for power users

A second tier of env vars that show up in real tuning playbooks.

NCCL_CHANNELS_PER_PEER

Number of parallel channels NCCL uses per peer connection. Default 1-2 depending on topology. Bumping to 4-8 increases bandwidth on high-port-count IB at the cost of more GPU memory for NCCL buffers. Useful on Quantum-2 / Quantum-3 NDR/XDR fabrics where a single channel under-utilizes 400-800 Gbps links.

NCCL_MIN_NCHANNELS / NCCL_MAX_NCHANNELS

Lower and upper bounds on the channel count NCCL picks. Setting NCCL_MIN_NCHANNELS=4 forces at least four parallel rings on every collective; useful for bandwidth-bound workloads. Setting NCCL_MAX_NCHANNELS=2 reduces memory and overhead for latency-bound workloads. Default range is 2-32 depending on topology.

NCCL_CROSS_NIC

Controls whether NCCL allows traffic between different NICs on the same node. Default 0 (no cross-NIC). Set to 1 on rail-optimized fabrics where cross-NIC traffic can offload congested rails — but only after benchmarking; it can also hurt if PFC is misconfigured.

NCCL_IB_QPS_PER_CONNECTION

Number of IB Queue Pairs per peer connection. Default 1. Increasing to 4 enables multi-path routing across IB and can extract more bandwidth from adaptive-routing-capable fabrics. Each QP costs ~1 MB of memory per peer; at 1024 ranks, four QPs use ~4 GB per rank.

NCCL_IB_SPLIT_DATA_ON_QPS

Splits a single collective's data across the QPs above. Enable (=1) only with NCCL_IB_QPS_PER_CONNECTION>1 and only on fabrics with reliable in-order delivery — out-of-order delivery causes catastrophic slowdowns when this is enabled.

NCCL_P2P_LEVEL

Granularity of P2P enable. NVL = NVLink only, PXB = PCI bridge OK, SYS = system memory OK, PHB = same PCI host bridge. Default is auto. Forcing NVL is a fast diagnostic: if performance degrades, NCCL was using PCIe paths you didn't realize existed.

NCCL_ASYNC_ERROR_HANDLING

When =1, NCCL surfaces collective failures asynchronously so the framework can tear down cleanly instead of hanging. PyTorch 2.0+ sets this by default; verify with python -c "import os; print(os.environ.get('NCCL_ASYNC_ERROR_HANDLING'))" if you suspect a hang isn't being detected.

Env var quick reference

Variable	Default	When to tune
`NCCL_DEBUG`	WARN	INFO when diagnosing, TRACE for deep debug only
`NCCL_TIMEOUT`	1800s	Lower in staging (300s), raise for long collectives
`NCCL_P2P_DISABLE`	0	Never set to 1 in production
`NCCL_IB_HCA`	auto	Set explicitly for multi-NIC rail-optimized
`NCCL_IB_GID_INDEX`	0	3 for RoCE v2 / IB; check `show_gids`
`NCCL_NET_GDR_LEVEL`	auto	5 to force GDR, verify with INFO logs
`NCCL_BUFFSIZE`	4 MB	8-16 MB for large-message workloads
`NCCL_NSOCKS_PERTHREAD`	2	4-8 on high-throughput IB
`NCCL_ALGO`	auto	Ring/Tree/NVLS to override
`NCCL_PROTO`	auto	LL/LL128/Simple — rarely needed
`NCCL_TREE_THRESHOLD`	~64 KB	Lower forces more Tree, raise forces more Ring
`NCCL_COLLNET_ENABLE`	0	1 to enable SHARP on Quantum IB
`NCCL_NVLS_ENABLE`	1	0 to disable in-switch reduction
`NCCL_CHANNELS_PER_PEER`	1-2	4-8 for high-BW IB
`NCCL_MIN_NCHANNELS`	2	Raise for bandwidth, lower for latency
`NCCL_CROSS_NIC`	0	1 on rail-optimized only after benchmarking
`NCCL_IB_QPS_PER_CONNECTION`	1	4 for multi-path adaptive routing

NCCL extended FAQ

Additional questions readers keep asking.

Q: How do I tell whether NVLS is actually engaged?

Set NCCL_DEBUG=INFO and look for log lines containing NVLS in the algorithm/protocol selection. The line typically reads NCCL INFO Channel xx: NVLS or NCCL INFO Algorithm/Protocol: NVLS/Simple. If you see only Ring/Simple or Tree/LL for an 8-GPU NVSwitch node on medium messages, NVLS isn't engaged — check NCCL version (2.16+) and that NCCL_NVLS_ENABLE isn't set to 0.

Q: What's the right NCCL_BUFFSIZE for 70B FSDP?

Start at 8 MB. Profile: if nccl-tests reduce_scatter_perf shows busbw climbing through 64 MB messages but plateauing earlier than NVLink can support, raise to 16 MB. The cost is NCCL_BUFFSIZE × channels × peers per rank — on 8-way TP that's ~512 MB at 16 MB. For 405B models, 32 MB is common.

Q: Can I run NCCL across IB and RoCE simultaneously?

Not in a single communicator. NCCL can address multiple HCAs of the same transport (set NCCL_IB_HCA to a list) but not mixed transports. The workaround is process-group splitting: one group for IB-connected ranks, one for RoCE-connected, app-level routing between. Rare in practice; most clusters are homogeneous.

Q: Why does my single-node 8-GPU AllReduce vary by 20% run-to-run?

Three usual suspects. First, NVLS engages above a message-size threshold and your messages straddle it — try NCCL_NVLS_ENABLE=1 to force. Second, NVSwitch routing has slight variability under contention; run nccl-tests with -c 1 (correctness check + warmup) and report only steady-state. Third, CPU affinity is wrong and NCCL's host threads bounce between cores — pin with numactl --cpunodebind or NCCL_IGNORE_CPU_AFFINITY=0.

Q: What's the practical maximum cluster size for NCCL in 2026?

NVIDIA has demonstrated 100k+ GPU NCCL deployments (xAI Colossus, Meta Llama-4 cluster). The bottleneck above ~16k GPUs is not NCCL itself but the subnet manager, the SHARP aggregation tree depth, and operational issues (per-day GPU failure count exceeds checkpoint frequency). NCCL scales; the cluster around it is what struggles.

Q: Why do my NCCL collectives get slower in the second hour of training?

Three causes ranked by frequency. First, thermal throttling — GPUs hit their TJ limit and clock down; NVLink bandwidth drops with GPU clock. Second, IB error counters climbing — bad cables or marginal optics cause retransmissions; check ibcheckerrors periodically. Third, memory fragmentation — long-running PyTorch allocator state grows; NCCL buffers re-allocate at worse addresses for DMA. Restart the process to confirm.

Q: Should I enable PXN (PCIe X-NIC)?

PXN routes intra-node traffic across NICs to reach the same destination via multiple paths. On rail-optimized 8-NIC nodes, set NCCL_PXN_DISABLE=0 (default in NCCL 2.18+) — it can add 10-20% on AllReduce when rail contention exists. Disable only if you observe PCIe contention with other workloads on the same node.

Q: How do I debug "NCCL WARN Call to ibv_modify_qp failed"?

The InfiniBand stack rejected NCCL's queue pair transition, usually because of a routing problem. Check that NCCL_IB_GID_INDEX matches the GID type your fabric uses (show_gids to enumerate). If the fabric is RoCE v2, you need a v2 GID; using a v1 GID will fail this exact call. Less commonly, the subnet manager is down — ibstat should show Active and LinkUp.

Q: Does SHARP work on virtualized / cloud IB?

Only when the cloud provider exposes SHARP-capable Quantum switches and runs the SHARP aggregation daemon. CoreWeave, Lambda Cloud, and on-prem clusters typically do. Most generic clouds (AWS, GCP) do not — they offer RoCE or EFA, which have no SHARP equivalent. Verify with sharp_cmd or check NCCL_DEBUG=INFO for CollNet algorithm selection.

Q: What's the relationship between NCCL and GPUDirect Storage?

Separate technologies. NCCL handles GPU-to-GPU collective traffic via GPUDirect RDMA (NIC ↔ GPU). GPUDirect Storage (GDS) handles NVMe ↔ GPU direct DMA, used for checkpoint loading and dataset streaming. They share the GPUDirect kernel module but are otherwise independent — NCCL doesn't move data to or from storage.

Q: Can NCCL use NVLink Switch (the standalone product) vs in-baseboard NVSwitch?

The external NVLink Switch (e.g., GB200 NVL72 spine) and the on-baseboard NVSwitch (HGX H100) look the same to NCCL — both expose the NVLink fabric topology to the driver. NCCL discovers via nvidia-smi nvlink -s equivalents at init. The only practical difference is NVL72 exposes a 72-GPU NVLink fabric, enabling TP=72 inside one collective domain.

Q: My nccl-tests busbw matches expectations but real training is still slow. Why?

Six things to check, in order. (1) Compute-collective overlap is broken — profile with Nsight, look for compute idle during collectives. (2) DDP bucket size is wrong — too small means too many small collectives. (3) FSDP is not prefetching the next layer's AllGather. (4) A straggler rank is stretching every collective to its slowest participant — collect per-rank step times. (5) Optimizer step is serialized after AllReduce when it could overlap. (6) Python is the bottleneck — switch on NCCL_LAUNCH_MODE=GROUP and CUDA Graphs.

Q: What's `NCCL_IB_TIMEOUT` and when should I raise it?

The number of 4.096 µs × 2^timeout units before IB declares a link unresponsive. Default 20 = 4.3 seconds. Raise to 22 (~17 seconds) on noisy RoCE fabrics where transient PFC pauses can exceed default. Don't raise above 24 — at that point you're hiding actual fabric issues that should fail loudly.

Q: Does NCCL work with MIG (Multi-Instance GPU)?

Yes, but each MIG slice is its own NCCL endpoint with its own NVLink visibility. MIG slices on H100 lose NVLink (NVLink is allocated to the full GPU), so MIG + NCCL means PCIe-only intra-node — useful for testing, not for production training. For tenanted inference at MIG granularity, use one MIG slice per inference replica and avoid multi-MIG TP entirely.

Q: How does NCCL handle Multi-Process Service (MPS)?

MPS lets multiple processes share a single GPU's CUDA contexts. NCCL works under MPS but you must export CUDA_MPS_PIPE_DIRECTORY=... consistently and ensure each NCCL rank gets its own SM partition. Not commonly used in production training; appears mostly in evaluation harnesses that share GPUs across short jobs.

Q: What's NCCL's behavior under preemption (SIGKILL on one rank)?

The killed rank's TCP connections close; surviving ranks see EPIPE on their next collective and either hang (without NCCL_ASYNC_ERROR_HANDLING=1) or raise (with). For graceful restart, frameworks call destroy_process_group() and init_process_group() again. Spot-instance training playbooks rely on this — see checkpoint storage and recovery for the wraparound.

Q: How does NCCL interact with Slurm's MPI plugins?

Slurm's --mpi=pmix or --mpi=pmi2 launches your job and sets up the process group. NCCL initializes inside that, using the rendezvous information Slurm provides (via env vars like SLURM_PROCID). PyTorch's torchrun and Slurm both work; the trick is making sure they don't fight over rank assignment. For Slurm-native: use srun directly and set MASTER_ADDR/MASTER_PORT from SLURM_* vars. See distributed training for full recipes.

Q: Are there NCCL benchmarks I should run on a new cluster acceptance test?

Yes — standard acceptance battery: (1) all_reduce_perf -b 8 -e 8G -f 2 on every GPU subset of interest (TP=8, full cluster). (2) alltoall_perf if you're running MoE. (3) A loopback iperf3 over IB to verify the fabric. (4) ib_send_bw -d mlx5_0 -F per HCA. (5) A 1-hour soak test to catch thermal and intermittent issues. Record baseline; re-run quarterly. Anything more than 5% regression is a ticket.

Q: Why does NCCL_DEBUG=INFO show "Trees [0] ..."?

The Trees [N] lines enumerate NCCL's chosen tree topologies for the channel — each channel can have a different tree structure to balance load. Multiple lines mean NCCL built multiple parallel trees and is using them in rotation. Normal and healthy; only worry if you see only one tree on a multi-rail fabric.

Q: Can I run NCCL inside Kubernetes without privileged containers?

Yes, with the NVIDIA device plugin and the network operator handling IB device exposure. The pod needs rdma/hca: 1 resource requests for IB visibility. Cilium and Calico CNI both support RDMA passthrough with the right configuration. Performance is identical to bare-metal once IB devices are exposed — there's no hypervisor in the data path.

Q: How do I prevent NCCL from grabbing all NICs on a multi-tenant node?

Set NCCL_IB_HCA=mlx5_0,mlx5_1 to restrict NCCL to specific HCAs, leaving others for other workloads. Combined with QoS configuration on the switch (traffic class via NCCL_IB_TC), you can co-host NCCL training with non-NCCL workloads on the same node without bandwidth contention. Rarely a good idea in production training, but useful in dev environments.

Q: What happens if I run NCCL with mismatched GPU counts per rank?

Undefined. NCCL assumes one CUDA device per rank by default. If you set CUDA_VISIBLE_DEVICES=0,1 on rank 0 and CUDA_VISIBLE_DEVICES=0 on rank 1, the rank-0 process will pick which GPU to use, but collectives won't form a coherent topology. Always use one GPU per rank; let the framework handle multi-GPU within a process via DDP/FSDP.

Q: Does NCCL handle GPU clock changes mid-collective?

Yes, but performance dips. If a GPU drops to a lower clock (thermal or power throttling) during a collective, that rank slows down and stretches the entire collective to its pace. NCCL does not adjust algorithm choice in flight. The fix is upstream: solve the thermal issue, not the NCCL configuration.

Q: What's NCCL's roadmap for UEC (Ultra Ethernet Consortium)?

NVIDIA participates in UEC but is hedging — Spectrum-X is NVIDIA's UEC-aligned Ethernet stack for AI, and NCCL has experimental Spectrum-X support since 2.22. The thesis: UEC standardizes lossless Ethernet semantics so collectives can run as well on Ethernet as on IB. Practical impact in 2026: limited; IB still wins on latency. By 2027-2028, UEC-compliant 800G Ethernet may equal IB for AllReduce.

Q: How do I know if `NCCL_LAUNCH_MODE=GROUP` is helping?

Profile with Nsight Systems and look at the CPU thread doing CUDA dispatch. GROUP mode batches NCCL launches so the CPU does fewer dispatches per step. The win is on workloads with many small collectives (FSDP, MoE) — expect 10-20% step-time improvement. Workloads with a few large collectives (DDP on big bucket sizes) see no benefit.

Q: Does NCCL respect CUDA streams set by the caller?

Yes. Every ncclAllReduce call takes a cudaStream_t argument. PyTorch wraps this via its stream API. Custom CUDA code can have NCCL run on a non-default stream, enabling overlap with compute kernels on other streams. This is how compute-collective overlap works in practice — different streams, queued in parallel, dispatched concurrently on the GPU's copy/compute engines.

Q: Can NCCL be replaced by gloo for inference?

In principle yes — init_process_group(backend="gloo") would work for TP inference. In practice no — Gloo's GPU collective is ~10× slower than NCCL and would dominate per-token latency. Don't do this except when debugging on a no-NCCL machine.

Q: What's the worst NCCL misconfiguration you've seen?

A toss-up between (a) NCCL_P2P_DISABLE=1 left in a Docker image and silently routing all 8-GPU traffic through host memory, costing 90% of NVLink bandwidth, and (b) a wrong NCCL_IB_GID_INDEX on a RoCE fabric causing NCCL to fall back to TCP and turning a 400 Gbps fabric into a 10 Gbps fabric. Both took weeks to diagnose because nothing fails loudly — performance just degrades. The fix in both cases is to make nccl-tests a startup gate: if busbw is below baseline, refuse to start training.

Q: How do I tell what NCCL version a running training job is using?

Set NCCL_DEBUG=VERSION and look at the first lines of stderr. Output looks like NCCL version 2.22.3+cuda12.4. In Python, torch.cuda.nccl.version() returns a tuple. In a Conda environment, conda list nccl or inspecting the PyTorch wheel metadata (pip show torch) shows the bundled version. Pin the version in your environment lock; don't rely on PyTorch's default to be consistent across releases.

Q: My nccl-tests numbers are good but real training is slow. What gives?

Three common reasons. First, nccl-tests runs in isolation; real training has compute-collective contention on the GPU's copy/compute engines. Second, nccl-tests uses one collective at a time; real training has many. Third, real training has data-loading and other CPU work that doesn't appear in the isolated benchmark. Profile with NSight Systems to see the timeline — most of the time the gap is "the GPU is doing other work and can't run the collective at peak rate."

Q: Is RDMA-over-Ethernet (RoCE v2) really equivalent to InfiniBand for NCCL?

Equivalent in principle (same RDMA semantics), different in operational pain. RoCE v2 needs PFC (Priority Flow Control) and ECN (Explicit Congestion Notification) configured correctly across the entire fabric to avoid packet drops; misconfiguration causes silent slowdowns. InfiniBand handles this in hardware. Production teams running RoCE v2 typically invest more engineering effort in fabric tuning. Performance can match IB; the operational complexity is higher.

Q: How do I detect a straggler rank?

Two signals. (1) Per-rank collective latency exceeds the cluster median by 2σ — your slowest rank is consistently slow. (2) Wall-clock step time correlates with one specific rank's CUDA timeline (NSight). Common causes: thermal throttling on that GPU, a NIC at half PCIe width, a noisy-neighbor on a shared node. Run nccl-tests with --allocator vmm against each rank to baseline; the outlier reveals itself.

Q: Does NCCL benefit from CPU affinity tuning?

Yes, slightly. NCCL's host-side threads (proxy threads, init threads) benefit from being pinned to the same NUMA node as the GPU's PCIe root complex. Most production stacks set OMP_PROC_BIND=close and OMP_PLACES=cores. Specific gains are usually 1–3% step-time; not the first lever to pull but a free improvement once everything else is tuned.

Q: How do I think about NCCL when designing a new cluster?

Three rules. (1) Match NIC count to GPU count for rail-optimized topology (one NIC per GPU, separate rails). (2) Pick a fabric (IB / RoCE / EFA / Slingshot) and stick with one — mixing creates configuration headaches. (3) Plan for SHARP if you're on Quantum-2/3 IB; the in-network reduction is meaningful at scale. The cluster topology will outlive multiple NCCL versions; design for the next 3 years of workloads, not just current.

Q: What's the realistic ceiling on NCCL scaling in 2026?

For training: 16k–32k GPUs with proper hierarchical communication (intra-node NVLink, inter-rack IB/EFA, with optional SHARP). Above that, you're typically using DiLoCo-style methods or mixture-of-experts patterns that reduce the synchronous AllReduce burden. For inference: TP=8 within a node is the practical ceiling for most production deployments; TP=16 across two nodes is possible but rarely worth the latency cost over running two TP=8 replicas.

Q: Should I care about NCCL on B300 / Rubin generation?

Yes — by mid-2026 some hyperscalers are deploying B300 (refreshed Blackwell) and prepping for Rubin (NVIDIA's 2026–2027 generation). Topology assumptions shift: NVL576 prototypes connect more GPUs in a single NVLink domain; SHARP variants evolve; NCCL versions track. Stay on the latest stable NCCL for new hardware; pin to known-good versions for stable workloads.

Q: Is the NCCL_ALGO knob worth overriding in production?

Rarely. The defaults are good. The cases where overriding has paid off in documented production deployments: (a) forcing Tree for very-small-message AllReduce in inference servers, (b) forcing Ring on certain EFA configurations where Tree was misbehaving, (c) forcing NVLS off when buggy on a specific NCCL release on prerelease Blackwell silicon. In all three the override was a temporary workaround until a NCCL patch shipped. Avoid making it permanent without justification.

Real-world NCCL failure modes

A taxonomy of how NCCL fails in production, with diagnostic signatures and root cause patterns.

The slow-straggler pattern

One rank consistently lags by 5-50 ms per collective. Cluster-wide step time is dictated by it. Root causes: a single GPU with degraded NVLink (one lane out of four down — nvidia-smi nvlink --status), a CPU thread pinned to a bad core (NUMA imbalance), a NIC firmware issue dropping ~1% of packets and forcing retransmits. The diagnostic signature is consistent across runs — same rank ID, same delay. Mitigation: replace or quarantine the GPU/NIC; in the interim, exclude the node from the job's host list.

The mystery hang

Training proceeds normally for hours, then every collective stops with no error logged. Almost always one of three things: (1) IB subnet manager restarted and the fabric is briefly partitioned, (2) a memory leak in user code triggered cudaMalloc to block and stall the NCCL kernel queue, (3) the kernel's OOM killer terminated one rank without notifying the framework. Set NCCL_ASYNC_ERROR_HANDLING=1, NCCL_TIMEOUT=1800, and TORCH_NCCL_BLOCKING_WAIT=0 to surface these as exceptions instead of hangs. Always run with dmesg | tail -100 ready for forensic inspection.

The cold-start cliff

First 10-100 collectives after job start are 2-5× slower than steady state. Causes: NCCL is profiling the topology and selecting algorithms (one-time cost), IB QPs are being allocated, NCCL channels are being warmed. Solution: include 50-100 warmup steps before measuring throughput. Production training loops do this implicitly; benchmarks must do it explicitly.

The version-drift slowdown

A previously fast cluster gets 10-20% slower after a routine PyTorch upgrade. Cause: new PyTorch shipped a new NCCL that changed default algorithm selection thresholds. Diagnostic: pin the old NCCL via LD_PRELOAD and re-benchmark. Permanent fix: identify the new threshold and adjust env vars (NCCL_TREE_THRESHOLD, NCCL_ALGO) or accept the new default if it's better on average.

The NIC failover anomaly

On rail-optimized clusters, when one rail's switch hiccups and PFC quiesces traffic, NCCL's adaptive routing should reroute via other rails. Sometimes it doesn't, and one rail's GPUs run at 10% of expected bandwidth until the job restarts. Mitigation: set NCCL_IB_AR_THRESHOLD=8192 (enable adaptive routing for messages >8 KB), monitor per-rail error counters, and have an auto-restart trigger when collective time variance exceeds 30% for 60 seconds.

The MTU-mismatch silent slowdown

NICs are configured for 9000-byte jumbo frames but one switch in the path is configured for 1500. NCCL works but every large message is fragmented and reassembled, losing 30-50% bandwidth. Detection: ping -M do -s 8972 from each rank to every other; failures indicate path MTU is below 9000. Fix is operational: every device along the path needs identical MTU.

The thermal cascade

In a hot data hall, GPUs throttle and slow down. Slow GPUs slow collectives. Slow collectives extend step time. Extended step time means GPUs are computing-active longer per checkpoint interval. GPUs get hotter. The whole cluster spirals into a thermal stable point at 70-80% of peak performance. Solution is cooling, not NCCL — but nvidia-smi --query-gpu=temperature.gpu,clocks.gr --format=csv -l 60 should be in your monitoring.

NCCL for inference at scale

Most NCCL writing focuses on training. Inference has different patterns that change what tuning matters.

Decode-phase TP all-reduce: latency over bandwidth

Each decoded token in TP=8 inference issues one AllReduce per transformer layer's attention output and one per MLP output. For Llama 70B with 80 layers, that's 160 collectives per token. Each operates on small tensors (hidden_dim × dtype_bytes ≈ 16 KB at hidden_dim=8192, BF16). Tree algorithm dominates Ring at this size; per-collective latency is 5-10 µs on NVLink, so 160 collectives add ~1-1.5 ms per token — a real fraction of the 20-30 ms inter-token latency for a 70B model.

Prefill-phase TP all-reduce: bandwidth over latency

Prefill processes the entire prompt in parallel. The same 160 collectives per layer now operate on hidden_dim × prompt_length × dtype_bytes — for an 8 K prompt at hidden_dim=8192, that's ~128 MB per collective. Ring + Simple wins here; total prefill collective time is ~500-800 µs per layer. This is why prefill is bandwidth-bound and decode is latency-bound — see disaggregated inference for the architectural implications.

CUDA Graph capture for inference

vLLM, TensorRT-LLM, and SGLang all capture decode-phase NCCL calls into CUDA Graphs. The graph replay amortizes per-call CPU overhead from ~10 µs to ~1 µs per collective. For 70B TP=8 at 50 tokens/sec, that saves ~70 ms/sec — material. Tradeoff: graphs lock the call sequence, so dynamic shapes (variable batch sizes) defeat capture. Modern serving frameworks bucket batch sizes and capture one graph per bucket.

Persistent communicators

Inference servers initialize NCCL once and reuse the communicator forever — unlike training, where preemption may force reinitialization. This shifts the cost calculus: spend longer on init (better topology probing, longer warmup) to win on every subsequent token. NCCL_GRAPH_REGISTER=1 (NCCL 2.20+) lets the framework register buffers once and avoid per-collective registration overhead.

Multi-replica serving

When running multiple replicas of a TP=8 model on a 64-GPU node (e.g., 8 replicas × TP=8), each replica should use a non-overlapping set of GPUs. NCCL communicators don't share state across replicas, but they share NVSwitch fabric bandwidth. Schedule replicas to non-adjacent GPU sets when possible to minimize NVSwitch contention.

NCCL version-by-version feature timeline

Tracking which NCCL release introduced which feature matters because Blackwell- and Grace-Hopper-class clusters need recent versions, and production teams routinely run mixed versions in long-lived clusters.

Version	Released	Notable additions
2.18	early 2023	LL128 protocol improvements, better PCIe topology detection
2.19	mid 2023	Initial SHARP v3 integration on Quantum-2
2.20	Q1 2024	User-buffer registration (`ncclMemAlloc`), CUDA Graph improvements, larger channel counts
2.21	Q3 2024	NVLS-Sharp (NVSwitch + Quantum-2 hybrid), improved EFA support, performance fixes for B200 prereleases
2.22	Q4 2024	Blackwell NVL72 awareness, NVLink-5 path detection, expanded Profiler API
2.23	Q1 2025	NCCL-Tests v2.23 alignment, RAS (Reliability/Availability/Serviceability) subsystem for hang detection
2.24	Q3 2025	NVL72-tuned NVLS variants, GB200 rack-aware topology hints, async error handling refresh
2.25	Q1 2026	Improved Quantum-3 SHARP, multi-rail EFA hints, Blackwell B300 prep

In production: pin a NCCL version per job; never let workers pull mismatched versions. NCCL 2.22+ is the floor for Blackwell deployments; 2.20+ for Hopper. For Volta-only clusters (still in service at some hyperscalers), 2.19 is the last fully-supported branch.

Mixed-version pain

A NCCL communicator is established from the version compiled into each worker's PyTorch / TRT-LLM / JAX binary. Mixing 2.20 and 2.22 workers in the same job is documented to cause silent slow paths and occasional hangs. Pin NCCL via a Conda lock file or a container image; verify with NCCL_DEBUG=VERSION at startup.

NCCL on AWS EFA SRD: the production reality

AWS Elastic Fabric Adapter (EFA) is the network for p5, p5e, p5en, and p6 instances (Hopper + Blackwell). EFA uses SRD (Scalable Reliable Datagram), a custom protocol that differs materially from InfiniBand.

Key points for NCCL operators:

No GID configuration. EFA doesn't use IB GIDs; the AWS plugin handles addressing. NCCL_IB_GID_INDEX is irrelevant; setting it is a tell that the team copied an on-prem playbook without adapting.
Use aws-ofi-nccl plugin. The aws-ofi-nccl plugin bridges NCCL to libfabric/EFA. Without it NCCL falls back to TCP — 10–20× slowdown. Verify with NCCL_DEBUG=INFO: look for NET/OFI Selected Provider is efa in the log.
Multi-rail. p5.48xlarge has 32× 100 GbE EFA NICs (3.2 Tbps total). NCCL must use all of them; the plugin handles striping. NCCL_NSOCKS_PERTHREAD and NCCL_SOCKET_NTHREADS are not the right knobs here — the plugin manages multi-rail under libfabric.
Topology hints. AWS publishes topology files for the larger instance types via /opt/amazon/efa/share/topology/. Point NCCL at them with NCCL_TOPO_FILE.
Placement groups. Cluster placement groups colocate instances on the same spine. Without one, inter-node latency varies wildly. Always use cluster placement for training jobs.
CloudWatch metrics. EFA exposes per-NIC counters via CloudWatch. Track RDMAWriteBytes, RDMAReadBytes, and packet-loss counters during nccl-tests baselines.

EFA performance benchmark on p5.48xlarge

A tuned 64-GPU job (8× p5.48xlarge, 8× H100 each) achieves roughly:

Intra-node AllReduce (8× H100, NVLink): 380 GB/s busbw
Inter-node AllReduce (8 nodes, EFA): 290 GB/s busbw on 1 GB messages
Latency, small messages: 8–14 µs intra-node, 24–35 µs inter-node

Performance below 250 GB/s on a tuned cluster is a sign of plugin misconfiguration, missing placement group, or noisy-neighbor contention.

NCCL on Slingshot 11 / Cray clusters

Slingshot 11 is HPE Cray's Ethernet-based fabric used in many DOE supercomputers (Frontier, El Capitan, Aurora). Slingshot is HPC-grade Ethernet — congestion management, adaptive routing, sub-µs switch latency — but it's not InfiniBand. NCCL integration goes through libfabric with the cxi provider.

For NCCL operators on Slingshot:

Use NCCL_NET_PLUGIN=ofi with the libfabric cxi provider.
HSN (High-Speed Network) tuning. Slingshot supports per-job HSN allocation; coordinate with the cluster scheduler (Slurm + plugin) to reserve fabric class.
No SHARP equivalent. Slingshot doesn't have in-network reductions. The fabric does have congestion management and adaptive routing, which helps multi-job tenancy.
Frontier-style topology. Frontier nodes have 4× MI250X (8 GCDs) per node connected via Infinity Fabric internally, with 4 Slingshot NICs per node. RCCL on MI250X / MI300 uses the same path through libfabric.

Production tip: HPE publishes a Cray Programming Environment with optimized libfabric and RCCL/NCCL configurations. Don't try to tune from scratch; start with the vendor's defaults.

NCCL_ALGO and NCCL_PROTO override patterns

NCCL's default algorithm and protocol selection is usually correct. The cases where overriding helps:

When to force NCCL_ALGO=Ring

Many GPUs per node (>16), large messages. Tree's logarithmic depth advantage shrinks; Ring's bandwidth advantage dominates.
EFA / TCP transports where Tree adds overhead. Tree's bidirectional pattern stresses some Ethernet fabrics.

When to force NCCL_ALGO=Tree

Small messages (<1 MB), many ranks (>64). Tree's O(log N) latency beats Ring's O(N).
Small inference servers using AllReduce for synchronization (not data movement).

When to force NCCL_ALGO=NVLS / NVLS+Sharp

8× H100/H200 single-node with NVSwitch. NVLS uses NVSwitch's multicast/reduction to do AllReduce in fewer hops than Ring. The default usually picks this automatically on supported hardware; verify with NCCL_DEBUG=INFO.
GB200 NVL72. NVLS-Sharp variants tuned for the 72-way fabric. NCCL 2.24+ required.

NCCL_PROTO=LL vs LL128 vs Simple

LL (Low Latency). Single 8-byte flits with embedded flag. Lowest latency; lowest bandwidth utilization. Good for tiny messages.
LL128. 128-byte flits with flag. The default for medium messages on NVLink.
Simple. No flag overhead; uses CUDA copy primitives. Best for large messages on PCIe / IB / EFA.

Override with NCCL_PROTO=Simple if you observe LL128 underperforming on very large messages — has been seen on B200 prereleases and some EFA configurations.

Topology override

NCCL_TOPO_FILE=path/to/topo.xml provides an explicit topology to NCCL. Useful when auto-detection misidentifies your fabric — common in containerized environments where PCI topology is opaque, or on heterogeneous clusters where some NICs are not visible to NCCL's probing.

The cluster-validation procedure: run nvidia-smi topo -m to see what CUDA sees; cross-reference with lspci and ibstat; if they disagree, build a topology file from lspci ground truth.

Inter-DC NCCL: DiLoCo and DiPaCo style training

A 2024–2026 trend worth tracking: distributed training across datacenters or geographies where the network latency between sites is too high for synchronous NCCL AllReduce. The motivating papers are DiLoCo (DeepMind, 2023) and DiPaCo (DeepMind, 2024) — gradient compression, local update accumulation, and periodic synchronization replacing per-step AllReduce.

The NCCL angle:

Intra-DC NCCL stays. Each datacenter runs standard NCCL with the local fabric (IB, EFA, Slingshot).
Inter-DC sync is custom. Custom collectives (often built on gRPC, Thrift, or specialized RDMA over WAN) handle the periodic gradient synchronization. NCCL isn't designed for >100 ms latency.
Gradient compression matters. PowerSGD, signSGD, or quantization to int8 reduces the inter-DC bytes by 4–32×. Pay quality cost; tune the compression ratio against your training stability.
Async semantics required. Inter-DC synchronization is asynchronous (workers don't block waiting for distant peers); the training algorithm must handle stale gradients.

Production deployments: rumored at scale at xAI (cross-region training of Grok 3+), Anthropic (cross-region for Claude post-training), Google (some Gemini training has cross-region components). Specifics aren't published; treat as an active engineering area rather than a deployed pattern.

Open-source efforts: Hivemind (Yandex / community), Prime Intellect's prime framework for decentralized training. Both abstract the inter-node communication as pluggable backends, with NCCL as the local one and custom transport as the wide-area one.

NCCL profiling and observability

For production training, NCCL's observability lags compute observability. The 2025–2026 improvements:

NCCL Profiler API (NCCL 2.22+)

Introduced a plugin interface for third-party profilers. Tools like NVIDIA NSight Systems, Hugo Profiler, and custom collectors can now see per-collective timing without modifying NCCL source. Use cases:

Identify slow ranks (stragglers).
Per-collective bandwidth tracking over a training run.
Detect channel-level imbalance.

RAS (Reliability, Availability, Serviceability) subsystem (NCCL 2.23+)

A built-in hang-detection mechanism. NCCL workers heartbeat; if one falls silent, the cluster's RAS service identifies the affected rank and emits a structured alert. Saves hours of "which node hung the training" debugging.

Enable with NCCL_RAS_ENABLE=1 and configure the RAS endpoint via NCCL_RAS_ADDR. Most cluster orchestrators (Slurm with NHC, Kubernetes with custom operators) now integrate RAS by default.

Per-rank metric export

Standard production patterns:

Export NCCL_DEBUG=INFO to per-rank log files; rotate hourly.
Parse the logs for NCCL Comm events to track communicator initialization patterns.
Use NSight Systems profiles on a sampled rank during baseline runs.
Pair with PyTorch's c10d traces for end-to-end visibility.

NCCL on Blackwell and GB200 NVL72

Blackwell introduces new collective primitives and the GB200 NVL72 changes the topology assumptions baked into older NCCL versions.

NVLink-5 bandwidth

Blackwell B200 has 1.8 TB/s of NVLink bandwidth per GPU (2× Hopper). A 1 MB AllReduce that takes 7 µs on H100 takes ~4 µs on B200 — proportional to the bandwidth doubling. Software-visible: nccl-tests reports ~1.5 TB/s busbw on 8× B200 single-node, vs ~750 GB/s on 8× H100.

Rack-scale NVLink (NVL72)

GB200 NVL72 connects 72 B200 GPUs via NVLink across the rack — no IB inside the rack. From NCCL's perspective, the rack is one giant single-node fabric. TP=72, EP=72, or PP=72 all execute as intra-fabric collectives. This is a meaningful architectural shift: workloads previously bound by inter-node IB now run at NVLink speed up to 72-way parallelism. See NVLink and rack-scale topology for the hardware story.

NCCL versions required

Don't run NCCL <2.22 on Blackwell. The pre-2.22 topology detection misidentifies NVL72's switch hierarchy and falls back to suboptimal Ring patterns. NCCL 2.23 (2025) added explicit NVL72 topology recognition. NCCL 2.24+ (2026) has tuned NVLS-Sharp variants for the 72-way fabric.

Multi-rack training on NVL72

Beyond a single rack, racks connect via 800G IB XDR (or Spectrum-X Ethernet). NCCL builds a two-level hierarchy: intra-rack NVLink (fast), inter-rack IB (slower). For a 4-rack 288-GPU job, expect intra-rack AllReduce ~3 ms for 1 GB, inter-rack AllReduce ~6 ms — much better than a flat 288-GPU IB topology would deliver.

2026-05-15 (v3): Expanded with algorithm bandwidth math, transport-layer comparison (NCCL vs UCX vs libfabric), framework-specific tuning (DDP/FSDP/Megatron/DeepSpeed/vLLM/JAX), determinism guide, NCCL-vs-TPU section, advanced env vars (channels, QPs, PXN), and 25 new FAQ entries.
2026-05-07 (v2): Complete-guide rewrite. TOC + 16 sections covering algorithms, protocols, env vars, IB/RoCE, debugging, common pathologies, FAQ.
2026-05-06 (v1): Original essay.

Table of contents

Key takeaways

Mental model: collective communication in one minute

The collective communication landscape in 2026

NCCL — NVIDIA Collective Communications Library

RCCL — ROCm Communication Collectives Library

oneCCL — Intel's Collective Communications Library

MPI — the classical HPC primitive

Gloo — PyTorch's CPU and fallback collective

SHARP — Scalable Hierarchical Aggregation and Reduction Protocol

PyTorch's c10d — the unified interface

JAX's pjit / XLA collectives

When each is the right choice

Library comparison table

Collective primitives explained: AllReduce, AllGather, ReduceScatter, Broadcast, AlltoAll

AllReduce

AllGather

ReduceScatter

Broadcast

AlltoAll

Send / Recv (point-to-point)

How frameworks compose them

Algorithm choices: Ring vs Tree vs CollNet vs Double Binary Tree

Ring

Tree (Binary / K-ary)

Double Binary Tree

CollNet

NVLS (NVLink SHARP)

Hierarchical (intra-then-inter)

How to know what was picked

Cross-vendor reality: NCCL vs RCCL vs MPI in 2026

NCCL on NVIDIA

RCCL on AMD

MPI as the universal fallback

The honest decision tree

A short history of NCCL

What NCCL actually does

Algorithms: Ring, Tree, NVLS

Ring

Tree

NVLS (NVLink SHARP)

How NCCL picks

Protocols: LL, LL128, Simple

Topology and how NCCL picks paths

Topology hints

Verification

Environment variables that matter

NCCL_DEBUG

NCCL_DEBUG_SUBSYS

NCCL_TIMEOUT

NCCL_P2P_DISABLE

NCCL_NET_GDR_LEVEL

NCCL_IB_HCA

NCCL_IB_GID_INDEX

NCCL_BUFFSIZE

NCCL_NSOCKS_PERTHREAD

NCCL_SOCKET_IFNAME

Putting it together

InfiniBand and RoCE

InfiniBand (IB)

RoCE (RDMA over Converged Ethernet)

Picking IB or RoCE

Multi-node training topologies

Rail-optimized topology

Fat-tree topology

NVLink islands within fat-tree

GB200 NVL72: rack-scale NVLink

Debugging hangs

First step: enable detailed logging

Common hang causes

Debugging command

Debugging slow collectives

Step 1: measure achieved bandwidth

Step 2: check algorithm choice

Step 3: check P2P

Step 4: check IB

Step 5: check for stragglers

nccl-tests: the diagnostic tool

Installation

Running all_reduce_perf