AI Cluster Networking: The Complete Guide — InfiniBand vs RoCE, Topology, Congestion Control

For frontier-scale training, the network is the bottleneck more often than the GPU is. A 16,000-GPU cluster spends a meaningful fraction of every step waiting on collectives — and the cost of getting any of it wrong scales with the dollar value of every wall-clock hour. Picking the right fabric (InfiniBand vs RoCEv2 vs AWS EFA vs Google Falcon), the right topology (rail-optimized fat-tree vs dragonfly vs HPN dual-plane), the right congestion control (DCQCN, HPCC, Swift, Falcon's hardware reliable transport), and the right optical layer (DAC vs AOC vs LPO vs co-packaged optics) is what separates clusters that train Llama-3 405B in 54 days from clusters that take 90.

This is the guide that ties all of those layers together. It is opinionated where the field has converged (rail-optimized topology, RDMA, jumbo frames, lossless Ethernet for RoCE) and explicit about where it hasn't (Ultra Ethernet vs InfiniBand long-term, CPO timelines, RoCE at >32k-GPU scale outside Meta). For the GPU-side mate of this guide — NVLink, NVSwitch, NVL72 and the in-rack fast fabric — read NVLink and rack-scale topology alongside.

Key takeaways
Mental model: AI cluster networking in one minute
The networking landscape in 2026
The networking layers
NVLink and NVSwitch
InfiniBand
RoCE (RDMA over Ethernet)
Topology: rail-optimized vs fat-tree
Bisection bandwidth: what it is, why it matters
Per-collective bandwidth requirements
Tail latency at scale
Diagnosing fabric health
GB200 NVL72: rack-scale NVLink
Cloud networking variants
Congestion control deep dive: DCQCN, HPCC, Falcon, SRD
Topology choices: fat-tree, rail-optimized, dragonfly, HPN
AI workload-aware routing
The bottom line
FAQ
Glossary
References
NCCL collective algorithms in depth (ring, tree, NVLS, SHARP)
Per-switch deep dive: Quantum-2/3, Spectrum-X, Tomahawk-4/5, Silicon One
Per-NIC deep dive: ConnectX-7/8, Cornelis, Pensando, EFA, gVNIC
Ultra Ethernet Consortium v1.0 and the 2026 timeline
DCQCN vs HPCC vs PFC tuning at frontier scale
LPO vs CPO economics and the 800G/1.6T transition
Dual-plane fabric designs (Meta, Microsoft, OpenAI patterns)
Reference designs at 1k, 8k, 32k, 100k GPUs
Failure-mode taxonomy and recovery patterns
Cross-DC training over WAN
Storage networking on the same fabric (NVMe-oF)
Per-cloud network reality (AWS EFA, Azure, GCP, Lambda, CoreWeave)
2026 frontier-cluster networking case studies

Key takeaways

Three layers of GPU networking, each with its own characteristics:

NVLink (within node): GPU-to-GPU within a server. 900 GB/s on H100, 1.8 TB/s on B200. Can't span nodes (except GB200).
InfiniBand or RoCE (between nodes): 200/400/800 Gb/s per port. RDMA-based. The fabric you build a multi-node cluster on.
Ethernet (general): management, storage, internet. Not for collectives.

Picking rules:

Single node, 8 GPUs: NVLink-only. Easy.
Multi-node, prioritize performance: InfiniBand 400 Gb/s or 800 Gb/s, rail-optimized topology.
Multi-node, prioritize flexibility: RoCE on lossless Ethernet.
Cloud: whatever your provider gives you (mostly RoCE on AWS, IB on dedicated clusters).
Frontier scale: GB200 NVL72 racks for the largest jobs.

The non-obvious thing: tail latency dominates throughput at scale. A 1% slow tail in collective time means 1% slower training. At $20M training runs, that's $200k.

Mental model: AI cluster networking in one minute

The problem has a name: the collective tail. Synchronous training is a barrier. Every GPU computes a gradient, every GPU joins an all-reduce, and nobody moves until the slowest GPU completes the collective. Mean bandwidth is a vanity metric — the step time is set by the p99 of the slowest link in the slowest rank. One congested switch port, one PFC pause storm, one ECMP collision, and a 16,000-GPU cluster waits. This is why "we have 800G everywhere" can still produce a 30% slower training run than a tuned 400G fabric: averages don't matter, tails do.

The fix is fabric engineering: deterministic latency, lossless transport, rail-optimized topology. InfiniBand wins by being designed for this from the wire up — credit-based flow control, in-order delivery, SHARP in-network reductions. RoCE wins by being cheap and ubiquitous if you configure PFC, ECN, DCQCN, and topology correctly. AWS SRD and Google Falcon win by giving up RDMA's reliable-connection model entirely and spraying packets across multiple paths to dodge hot spots. The analogy is highway design: a six-lane road with one accident is slower than a four-lane road with none, and the entire job of the network architect is to design out the accidents.

Without tail discipline vs with tail discipline:

Aspect	Untuned RoCE	InfiniBand / tuned fabric
Mean all-reduce latency	Low (looks fine)	Low
p99 all-reduce latency	3–5× worse	Tight to mean
Throughput at 16k GPUs	50–70% of theoretical	85–95% of theoretical
Loss recovery	TCP-style retries (slow)	Credit-based, near-lossless
Operational complexity	High (PFC, ECN, ECMP tuning)	Lower (vendor-defaulted)
When it pays off	Cost-sensitive, willing to tune	Frontier scale, can't afford tail

Production one-liner (NCCL): NCCL_IB_HCA=mlx5_0,mlx5_1,... to pin rails, NCCL_ALGO=Tree,Ring to choose collective shape, NCCL_PROTO=Simple,LL,LL128 to control the transport protocol.

Sticky number: RoCE p99 collective tail runs 3–5× worse than InfiniBand without disciplined PFC/ECN tuning — and that ratio is exactly what shows up in end-to-end training step time at >8k GPUs.

The rest of this guide is the layers, the topologies, the congestion-control algorithms, and the operational details that determine which side of that ratio you land on.

The networking landscape in 2026

The fabric market for AI training is now genuinely plural. Five years ago "AI cluster networking" effectively meant Mellanox InfiniBand HDR with NCCL on top; today the live options span at least four protocol families and three different transport philosophies.

InfiniBand (NVIDIA / ex-Mellanox): still the default for on-prem frontier clusters and most specialist GPU clouds. The current generation is Quantum-2 at 400 Gb/s NDR with 64 ports per 1U switch and SHARP in-network reductions; Quantum-3 at 800 Gb/s XDR is shipping into 2026 frontier deployments. Pair with ConnectX-7 HCAs (400 Gb/s) or ConnectX-8 (800 Gb/s) on the host side. Mature, deterministic, expensive, NVIDIA-locked.

RoCEv2 on lossless Ethernet: the cost-and-flexibility play, and now proven at hyperscale. Meta's RDMA over Ethernet for Distributed AI Training at Meta Scale (SIGCOMM 2024) is the canonical existence proof: 32k-GPU GenAI clusters built on RoCEv2 with custom rail-optimized topology, careful PFC/ECN tuning, and a receiver-driven congestion-control variant. Alibaba HPN (Qian et al., SIGCOMM 2024) is the other reference design, built around dual-plane Ethernet to eliminate single points of congestion.

AWS EFA (Elastic Fabric Adapter): AWS's user-space, OS-bypass transport over standard datacenter Ethernet. EFA uses SRD (Scalable Reliable Datagram) — a custom transport that does multipath spraying instead of relying on RDMA's reliable connection model. In P5/P5e/P6 instances it delivers 3.2–6.4 Tb/s of aggregate per-node bandwidth and is the standard fabric for AWS-hosted frontier training.

Google Falcon: Google's hardware-offloaded reliable transport, designed to give InfiniBand-class latency and loss recovery on standard Ethernet. Announced via the Google Cloud systems blog and contributed to the Open Compute Project. Falcon is now part of Google's strategy for Ultra Ethernet–style fabrics on A3/A4 GPU instances.

Microsoft Frontier Edge (FE): Azure's name for its purpose-built AI fabric on the ND-series GPU clusters — InfiniBand on the largest configurations, with custom topology and scheduling on top. Less publicly documented than EFA or Falcon but is the fabric behind Microsoft's frontier AI co-location with OpenAI.

Ultra Ethernet Consortium (UEC): industry-wide effort (AMD, Arista, Broadcom, Cisco, Google, HPE, Intel, Meta, Microsoft and dozens more — see ultraethernet.org) to standardize an Ethernet-based AI/HPC transport that competes with InfiniBand on tail latency and collectives, but with Ethernet's economics. UEC v1.0 specifications landed in 2024–2025; first compliant hardware shows up in 2026.

The optical layer: at 400G and 800G, the cable type matters as much as the switch. DAC (direct attach copper) is cheapest but capped at ~3 m. AOC (active optical cable) bridges 5–30 m at higher cost. LPO (linear-drive pluggable optics) is the 2025–2026 story: cuts power per port by 30–50% versus traditional retimed optics by removing the DSP. CPO (co-packaged optics) integrates the optics into the switch ASIC and is the path to >100 Tb/s switch radix; NVIDIA's Quantum-X Photonics and the Rubin platform start landing this in 2026–2027.

Quick-reference: protocols at a glance

Fabric	Max per-port BW (2026)	Switch-to-switch latency	Deployment	Who runs it at scale
InfiniBand NDR (Quantum-2)	400 Gb/s	~1–2 µs	On-prem frontier, specialist clouds	NVIDIA DGX SuperPOD, Microsoft Azure ND, CoreWeave, Lambda
InfiniBand XDR (Quantum-3)	800 Gb/s	~1 µs	2026 frontier, Blackwell-class	xAI Colossus 2, OpenAI Stargate, Anthropic
RoCEv2 on 400/800G Ethernet	400–800 Gb/s	~5–10 µs	Hyperscale and on-prem	Meta GenAI clusters (32k+ H100), Alibaba HPN
AWS EFA / SRD	400 Gb/s × 8 = 3.2 Tb/s/node	~15–25 µs (multipath)	AWS only	AWS P5/P5e/P6, Trainium UltraClusters
Google Falcon	400+ Gb/s	low single-µs target	GCP A3/A4, OCP-contributed	Google Cloud TPU and GPU clusters
Ultra Ethernet (UEC v1)	800 Gb/s	comparable to IB target	2026+ rollout	AMD/Broadcom/HPE-led ecosystem
NVLink 5 (intra-rack)	1.8 TB/s aggregate per GPU	sub-µs	NVL72 and HGX	Anyone running B200/GB200

NVLink is in the table as a reminder: it is the only "fabric" in the list that runs at near-HBM speed, and it is why every protocol decision above is really about what happens after you cross the rack boundary. Inside the rack you want NVLink (or, on AMD, Infinity Fabric / UALink); outside the rack you pick from the list above.

The networking layers

A frontier training cluster has three distinct networks:

Layer 1: intra-node (NVLink)

Within an 8-GPU server, all GPUs connect via NVLink. This is the fastest layer — 900 GB/s/GPU on H100, 1.8 TB/s on B200.

This is where TP collectives happen. TP=8 within one node uses NVLink exclusively.

Layer 2: inter-node (InfiniBand or RoCE)

Between nodes, RDMA-based fabric. 200, 400, or 800 Gb/s per port. Each node typically has multiple ports (rail-optimized).

This is where DP, PP, and (across nodes) FSDP collectives happen.

Layer 3: management plane (Ethernet)

Standard Ethernet for management, storage, monitoring. Not for collectives.

The takeaway: NVLink is for TP, InfiniBand/RoCE is for DP and PP, Ethernet is for everything else.

NVLink and NVSwitch

NVLink versions

Version	Per-GPU bandwidth	Used in
NVLink-3	600 GB/s	A100
NVLink-4	900 GB/s	H100, H200
NVLink-5	1.8 TB/s	B100, B200

Bidirectional. The numbers above are per-direction.

NVSwitch

NVLink fabric switch. Provides full all-to-all between connected GPUs.

DGX H100 NVSwitch: 8 GPUs all-to-all via internal NVSwitch. 900 GB/s/GPU sustained for collectives.
GB200 NVL72 NVSwitch: 72 GPUs all-to-all across one rack. 1.8 TB/s/GPU.

Why NVLink matters

For TP collectives (per-layer all-reduce of activations), NVLink is essential. Inter-node alternatives (IB, RoCE) are 10-20× slower.

A typical Llama-3 70B training step:

Per-layer activation all-reduce: ~256 MB.
80 layers × 2 (after attn, after MLP) = 160 collectives per forward+backward.
Total intra-node bandwidth used: ~40 GB.

On NVLink (900 GB/s): ~~50ms total. On IB 400 Gb/s (~~50 GB/s): 2.5 minutes. Not viable.

This is why TP almost never spans nodes. NVLink is required.

InfiniBand

InfiniBand (IB) is NVIDIA's preferred inter-node fabric. Native RDMA, low-latency, high-bandwidth.

Bandwidth tiers

HDR (200 Gb/s): 2018-era, still in many clusters.
NDR (400 Gb/s): 2022-era, current sweet spot.
XDR (800 Gb/s): 2024+, frontier deployments.

Per-rail vs per-rack bandwidth

A rail-optimized cluster has 8 rails per node. Bandwidth math:

HDR per node: 8 × 200 Gb/s = 1.6 Tb/s.
NDR per node: 8 × 400 Gb/s = 3.2 Tb/s.
XDR per node: 8 × 800 Gb/s = 6.4 Tb/s.

For frontier-scale training, NDR is the floor; XDR is the ceiling that justified investment.

Latency

IB latency is the strong point: ~1-2 microseconds switch-to-switch. RoCE adds ~5-10 microseconds for protocol overhead.

For collective small-message phases (initial reductions in tree algorithms), latency dominates throughput. IB wins.

Operational notes

IB drivers come from NVIDIA's MLNX_OFED package.
Switches are NVIDIA's Quantum (HDR/NDR) or Quantum-2 (XDR). See NVIDIA's product documentation for port counts and SHARP support.
Subnet manager (typically OpenSM) handles routing.

IB requires more specialized skills to operate than Ethernet, but the operational maturity is good — NVIDIA ships well-tested stacks. For the collective library that sits on top, see the NCCL guide and NCCL docs.

RoCE (RDMA over Ethernet)

RoCE provides the same RDMA semantics over Ethernet hardware. Two variants: RoCE v1 (rare, link-local), RoCE v2 (standard, routable).

Why RoCE

Reuses Ethernet infrastructure.
Cheaper switches (commodity, not specialized like IB).
Easier to operate for teams familiar with Ethernet.
Cloud providers increasingly offer it.

Why not RoCE

Ethernet is lossy by default. RoCE requires lossless Ethernet via Priority Flow Control (PFC). Misconfigured PFC = catastrophic NCCL slowdowns or deadlocks.
ECN (Explicit Congestion Notification) needs proper tuning — see DCQCN (Zhu et al., SIGCOMM 2015), the canonical congestion-control scheme for RoCEv2.
Multi-vendor switch ecosystems can have subtle interop issues. Meta's RDMA over Ethernet for Distributed AI Training at Meta Scale (SIGCOMM 2024) is the canonical writeup of how a hyperscaler operates RoCE at 32k-GPU scale.

When RoCE works well

Single-vendor Ethernet stack (Arista, Mellanox/NVIDIA, Cisco) with careful PFC config.
Cloud-managed (AWS, GCP) where the provider handles the lossless layer.
Cluster sizes < 10,000 GPUs where the operational complexity is manageable.

When RoCE struggles

Mixed-vendor environments. Subtle interop issues that surface as random NCCL slowdowns.
Frontier-scale (>10k GPUs). IB's operational maturity at that scale is currently unmatched.

Cloud RoCE

AWS: most large GPU instances use RoCE (EFA — Elastic Fabric Adapter). Performance is good in well-known configs.
GCP: similar pattern with Intel TPU/GPU instances.
Azure: InfiniBand on the dedicated AI clusters.
CoreWeave / Lambda / dedicated GPU clouds: typically InfiniBand.

Topology: rail-optimized vs fat-tree

Rail-optimized (NVIDIA's recommendation)

Each node has 8 NICs. 8 separate "rails" — independent network fabrics. Rail 0 connects all GPU 0s across nodes; rail 1 all GPU 1s; etc.

Within a rail, traffic is between same-position GPUs (e.g., GPU 0 on node A talks to GPU 0 on node B). NCCL routes collectives along these rails.

Pros: minimal cross-rail interference. Each rail can run at full bandwidth.

Cons: only useful for AI workloads (the 8-GPU-symmetry pattern). Less flexible for mixed traffic.

Fat-tree

Older topology. All nodes connect through a hierarchical tree of switches. Bisection bandwidth equals or approximates total per-leaf bandwidth.

Pros: flexible, well-understood, supports diverse workloads.

Cons: at frontier scale, requires very high switch counts to maintain bisection bandwidth.

Modern frontier clusters

Mostly rail-optimized. Llama-3's training cluster used rail-optimized topology with NDR InfiniBand.

For mixed-workload datacenters, fat-tree remains common.

Rail-optimized in practice: the 1024-GPU reference design

A canonical 1024-GPU rail-optimized cluster as of 2026: 128 DGX H100 nodes, each with 8 ConnectX-7 NICs at 400 Gb/s NDR. Eight independent rails. Each rail has 128 endpoints, so a single 64-port Quantum-2 leaf switch handles half a rail and pairs of leaves form a single rail's spine. Total switch count: 8 rails × 2 leaves × 1 spine pair = 24 switches plus inter-rail aggregation. Aggregate bisection: ~6.4 Tb/s per node × 1024 nodes / 2 = ~3.3 Pb/s. The cost: roughly $2-3M in switches and optics for this fabric, vs $40M+ for the 1024 H100 GPUs themselves. The network is 5-8% of cluster capex but 100% of the bottleneck for training step time.

Why dual-plane topologies became popular in 2024-2025

Alibaba HPN's "dual-plane" Ethernet design (SIGCOMM 2024) splits the fabric into two physically independent planes — each end host has NICs in both. The reason: ECMP hash-collision microbursts that cause 1-in-1000 collectives to stall at the 99.9th percentile. With dual planes, traffic can be load-balanced flow-by-flow across both planes, and a stuck plane affects only half the bandwidth. Meta's RoCE-at-scale paper makes a similar argument: at 32k GPUs you cannot avoid microbursts in any single-plane design, so you architect for plane-level fault tolerance from day one. NVIDIA's reference InfiniBand topology relies on SHARP (in-network reductions) plus adaptive routing to achieve similar resilience within a single plane.

Bisection bandwidth: what it is, why it matters

Bisection bandwidth: the minimum bandwidth crossing any plane that divides the network in half.

A network has good bisection bandwidth if any half can talk to the other half at full speed. This matters for collectives like all-to-all (MoE expert routing), which require full bisection bandwidth.

Per-rail bisection

In a rail-optimized cluster, each rail's bisection bandwidth is independent. If rail 0 has 1.6 Tb/s bisection, rail 1 also has 1.6 Tb/s independently.

Total cluster bisection: sum across rails. 8 rails × 400 Gb/s × 50% bisection efficiency ≈ 1.6 Tb/s effective per node.

When bisection matters

DP all-reduce: rail-local; bisection less critical.
EP all-to-all: requires full bisection (random routing patterns). Frontier MoE training pushes bisection limits.
CP ring attention: ring patterns; bisection less critical.

If you're training MoE at scale, bisection bandwidth is a primary concern. If you're training dense models, rail-bandwidth is the primary concern.

Per-collective bandwidth requirements

Quick reference for what bandwidth each collective needs:

Collective	Pattern	Bandwidth requirement	Latency requirement
TP all-reduce (small tensor)	All-to-all-reduce	Moderate (per-link)	Low
TP all-reduce (large tensor)	All-to-all-reduce	High (per-link)	Moderate
DP all-reduce	All-to-all-reduce	High (aggregate)	Moderate
PP send/recv	Point-to-point	Low	High
EP all-to-all	All-to-all	Full bisection	Moderate
FSDP all-gather	All-to-all-gather	High	Moderate
FSDP reduce-scatter	All-to-all-reduce	High	Moderate

The takeaway: dense LLM training (DP+TP) is bandwidth-heavy but latency-tolerant. MoE adds bisection-heavy all-to-all (see mixture-of-experts serving for the inference side). PP is latency-sensitive but low-bandwidth. For the parallelism choices that drive these collectives, see distributed LLM training.

Tail latency at scale

At 16,000 GPUs, every collective involves all 16,000 GPUs. The collective completes when the slowest GPU finishes. Slowest GPU = tail.

A 1% straggler — one GPU that's 1% slower — slows down the entire job by ~1%.

Sources of tail:

Hardware variance (one GPU thermal-throttles).
Network congestion (one rail has microbursts).
Garbage collection pauses (Python).
NUMA imbalances.

Mitigations:

Watch P99 collective time, not just average.
Replace consistently-slow nodes proactively.
Tune NUMA pinning.
Detect and skip stragglers via "dropping" mechanisms in some research stacks.

For frontier training, tail latency tracking is a first-class concern. Cluster operators monitor P99 of every collective every minute. The foundational treatment is Dean & Barroso's "The Tail at Scale" (CACM 2013) — written for web-serving but every word applies to collective-bound training. For the rack-scale view, see NVLink and rack-scale topology.

Diagnosing fabric health

nccl-tests

The canonical tool. See the NCCL guide for usage.

For multi-node fabric health, run all_reduce_perf across the entire cluster and compare to expected:

mpirun -np 64 -hostfile hosts.txt ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Expected (NDR 400 Gb/s, rail-optimized): ~40 GB/s busbw on 1 GB messages.

If you're getting <50% of this, troubleshoot.

IB diagnostics

ibstat            # link state per HCA
ibhosts          # discover IB topology
ibcheckerrors   # error counters across the fabric
ibhostd         # individual host diagnostic

Bad GID, bad cable, or congestion shows up here.

Continuous monitoring

For production clusters:

Per-port bandwidth utilization (Prometheus / Grafana).
Per-port error counters.
Latency histograms per node.
Collective time histograms (from training logs).

Alert on:

Bandwidth dropping below baseline.
Error counters increasing.
P99 collective time > 2× baseline.

GB200 NVL72: rack-scale NVLink

NVIDIA's GB200 NVL72 is a single rack with 72 Blackwell GPUs all connected via NVLink-5 + NVSwitch.

What changes

Pre-GB200, NVLink stopped at the 8-GPU node boundary. TP > 8 had to go via slower IB.

With GB200 NVL72, NVLink spans the entire 72-GPU rack. TP=72 or EP=72 within one fabric.

When this matters

Frontier training where the largest models can't fit in TP=8 efficiently.
MoE models with many experts that benefit from EP=64+.
Scale-out workloads that have outgrown 8-GPU islands.

When it doesn't matter

Most training in 2026 still uses 8-GPU TP groups within standard DGX. GB200 is overkill.
Inference workloads — TP=72 is rare for serving (latency cost of bigger collectives).

Cost

GB200 NVL72 racks cost millions of dollars and require liquid cooling. Frontier labs deploy them; everyone else watches.

Cloud networking variants

AWS

EFA (Elastic Fabric Adapter): AWS's RDMA-over-Ethernet. Used on P5 instances (H100), P4d (A100). Performance is good but not as predictable as dedicated IB.
Topology: rail-optimized within an availability zone.
Per-instance bandwidth: 400 Gb/s on P5.

GCP

vNIC (RoCE): on GPU instances.
A3 (H100) and A4 (B200) instances have high-bandwidth interconnects.
Cross-zone latency is significant; keep training within a zone.

Azure

ND series with InfiniBand on dedicated clusters.
Performance is competitive with on-prem IB.

Specialized GPU clouds (CoreWeave, Lambda, Crusoe)

Most use InfiniBand.
Tighter integration with NVIDIA's reference architectures (see NVIDIA datacenter GPUs).
Lower-cost than hyperscalers for dedicated long-running workloads. The decentralized end of the spectrum is covered in decentralized GPU compute.

Google's response to RoCE's operational complexity is Falcon, a hardware-offloaded reliable-transport protocol intended to give InfiniBand-class behavior on standard Ethernet.

Picking a cloud

For training workloads:

Bursty / experimentation: any major cloud is fine.
Long-running frontier training: dedicated GPU cloud or Azure ND. AWS works but watch performance variance.
Multi-region or hybrid: stay within one region; networking across regions is bad for training.

A short history of AI training networking

The networking requirements for AI training have evolved dramatically.

2010-2015 (pre-LLM): Ethernet was sufficient. Models trained on single GPUs or small clusters. Networking was an afterthought.

2016-2018: training spans multiple nodes. InfiniBand FDR (56 Gb/s) becomes standard for HPC-style AI clusters.

2019-2020: GPT-2 / GPT-3 training requires thousand-GPU clusters. EDR (100 Gb/s) IB is the workhorse.

2021: A100s with NVLink-3 drop intra-node communication cost. Inter-node moves to HDR (200 Gb/s).

2022: H100 clusters launch with NDR (400 Gb/s). Rail-optimized topology becomes standard.

2023: 1k-10k GPU clusters routine for frontier training. Per-node bandwidth: 8 × 400 Gb/s = 3.2 Tb/s.

2024: Llama-3 405B trained on 16k+ H100s with NDR. RoCE alternatives mature in cloud (AWS EFA, GCP).

2025: GB200 NVL72 launches — rack-scale NVLink fabric. Some training collapses inter-node communication into a single rack.

2026 (current): XDR (800 Gb/s) IB available. 100k-GPU clusters announced. Co-packaged optics in development.

The pattern: networking has scaled with model sizes. Today's frontier clusters use networking technology that didn't exist 5 years ago.

NCCL collective performance characteristics

For each major collective, the bandwidth math:

All-reduce (DP gradients, TP activations)

Move 2(N-1)/N × M bytes per rank, where M is message size and N is rank count.

For DP with 8 GPUs and 1 GB gradient: each rank moves ~1.75 GB. On NVLink (within node, 900 GB/s): 2ms. On NDR IB (across nodes, 50 GB/s effective): 35ms.

For TP=8 with 256 MB activation: each rank moves ~448 MB. On NVLink: ~0.5ms. Across nodes: 9ms.

All-gather (FSDP parameters)

Each rank receives the full reduced result. Bandwidth: M × (N-1)/N per rank.

Reduce-scatter (FSDP gradients)

Inverse of all-gather. Same bandwidth math.

All-to-all (MoE routing)

Each rank sends a different chunk to every other. Bisection bandwidth bounds it.

Broadcast

One sender, many receivers. Lower bandwidth requirement than all-reduce. Used at startup.

Tree vs Ring algorithms

NCCL picks based on message size:

Small messages: Tree (latency-optimal, O(log N) hops).
Large messages: Ring (bandwidth-optimal, O(N) hops with full bandwidth).

Crossover around 64 KB-1 MB depending on topology.

Topology choice in detail

The right topology depends on workload.

Rail-optimized

Each rank position has its own dedicated network rail.

Pros:

Per-rail can run at full bandwidth without contention.
Naturally fits TP=8 within node + DP across nodes pattern.
Standard in NVIDIA reference architectures.

Cons:

Specific to symmetric AI workloads.
Less flexible for mixed workloads.

Fat-tree

Hierarchical tree with full bisection bandwidth at each level.

Pros:

Flexible — supports any workload pattern.
Well-understood.

Cons:

More expensive at scale (more switches).
Less optimal for AI training specifically.

Dragonfly+

Modern hierarchical topology used in some HPC and AI clusters.

Pros:

Better path diversity than fat-tree.
Used in some top-500 supercomputers.

Cons:

More complex routing.
Less common in AI-specific deployments.

Cluster-of-clusters

Multiple smaller clusters connected loosely. Inter-cluster traffic is slower.

Used for: cost-sensitive deployments, regional partitioning, organizational separation.

Not ideal for synchronous training (cross-cluster collectives are slow). Works for federated learning or independent sub-jobs.

Per-node bandwidth math

Calculate aggregate cluster bandwidth:

For 8-GPU H100 node with 8 × 400 Gb/s NDR NICs (rail-optimized):

Per-NIC bandwidth: 400 Gb/s = 50 GB/s.
Per-node aggregate: 8 × 50 = 400 GB/s.

For an 1024-GPU cluster (128 nodes):

Aggregate bandwidth: 128 × 400 GB/s = 51.2 TB/s total.
Per-rail aggregate: 50 GB/s × 128 nodes = 6.4 TB/s per rail.

This dictates achievable collective throughput at scale.

For a 16k-GPU cluster (2k nodes):

Aggregate: 800 TB/s.
Per-rail: 100 TB/s.

These numbers are huge but they have to be — frontier training shovels petabytes of gradient data per training step.

Switch architectures

The switches themselves matter for network performance.

NVIDIA Quantum-2 (NDR)

NVIDIA's flagship IB switch. 64 × 400 Gb/s ports per 1U.

Features:

Adaptive routing.
Congestion control.
SHARP for in-network reductions.
Lossless fabric guarantees.

Used in DGX SuperPOD reference designs and most frontier AI clusters.

Quantum-3 (XDR)

Successor: 64 × 800 Gb/s ports. 2026+ deployments.

Cisco/Arista/Mellanox Ethernet for RoCE

For RoCE deployments, switch firmware matters:

PFC (Priority Flow Control) configuration.
ECN (Explicit Congestion Notification) tuning.
Support for lossless QoS classes.

Misconfigured switch firmware is a common cause of "RoCE works but at half the expected bandwidth."

NCCL on RoCE: gotchas

RoCE is more error-prone than IB. Common issues.

GID confusion

GID (Global ID) must match on both ends. Wrong GID = falls back to slower path.

# Check available GIDs
show_gids

# Often you want GID index 3 (RoCE v2)
export NCCL_IB_GID_INDEX=3

MTU mismatch

If switch MTU and host MTU differ, RoCE fragments. Slow.

Standard: 9000 bytes (jumbo frames) end-to-end.

PFC not propagated

If PFC is enabled on hosts but not on intermediate switches, lossless guarantees break. RoCE assumes lossless; without it, packet drops cause exponential backoff.

Buffer credits

RoCE switches need enough buffer for PFC pause times. Insufficient buffers = PFC storms.

For modern AI workloads, ensure switches have at least 10 MB shared buffer per port group.

Fabric monitoring in production

What to watch for healthy fabric.

Per-port metrics

Bandwidth utilization (should match expected for active workloads).
Error counts (should be near zero).
PFC pause time (RoCE) — high pause = congestion or misconfigured.
Discards (Ethernet) — should be zero.

Per-host metrics

IB link state (up, with correct speed).
HCA error counts.
Port speed (correct generation: HDR/NDR/XDR).

Periodic benchmarks

Run nccl-tests weekly to validate fabric performance hasn't degraded:

mpirun -np 64 -hostfile hosts.txt \
  ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8 \
  > weekly-fabric-check.log

Alert if achieved bandwidth drops below baseline by >10%.

Optical interconnect future

Co-packaged optics (CPO) are expected to change fabric design.

Why optics

Lower power per bit than electrical.
Higher density than coaxial.
Can carry signal kilometers (vs ~10m for electrical).

Today

Optical transceivers in pluggable modules (QSFP-DD, OSFP). Bandwidth scales nicely but power consumption is significant.

Co-packaged optics (CPO)

Optics integrated into the switch ASIC. Lower power, higher density.

NVIDIA's Rubin platform (2026-2027) includes early CPO. Replaces some pluggable optics with integrated.

By 2028-2029, expect CPO to be standard in frontier AI clusters. Reduces power consumption per bit by 30-50% and enables higher port densities.

Photonic compute

Even more speculative: optical compute (using light for matrix multiplications).

Companies like Lightmatter, Celestial AI exploring this. Production deployment is years out.

Congestion control deep dive: DCQCN, HPCC, Swift, Falcon, SRD

Once you accept that AI collectives generate the bursty, all-to-all, microsecond-incast traffic patterns that classic TCP was never designed for, the choice of congestion control becomes a primary lever — not a footnote.

DCQCN (the RoCEv2 baseline)

DCQCN (Zhu et al., SIGCOMM 2015) is the canonical congestion-control scheme for RoCEv2. It pairs ECN marking at switches with a rate-limited reaction at the NIC: when congestion is detected, the sender multiplicatively decreases rate; when ECN clears, the sender additively (then hyper-actively) increases. PFC sits behind it as the "don't drop" backstop; DCQCN exists precisely so PFC almost never has to fire.

DCQCN works, but tuning it is non-trivial. The four classic knobs (Rai, Rhai, Kmin, Kmax) interact with switch buffer depth and PFC headroom, and the wrong settings produce either silent PFC storms or chronic under-utilization. Most production RoCE clusters in 2026 ship DCQCN with vendor-tuned defaults (Mellanox/NVIDIA, Arista, Cisco), and operators tune from there.

HPCC (in-network telemetry)

HPCC (Li et al., SIGCOMM 2019) replaces DCQCN's coarse ECN signal with In-band Network Telemetry (INT): every switch on the path stamps the packet with its queue depth and link utilization, so the sender can compute a precise window directly from path state. The result is faster convergence and lower queueing — at the cost of needing switches that implement INT and NICs that act on it.

HPCC is the intellectual ancestor for most modern AI congestion-control proposals, and several hyperscaler-internal schemes are HPCC variants.

Swift and timely-style RTT control

Google's earlier work on Swift (and the older Timely) used RTT itself as the congestion signal, on the theory that RTT is a near-perfect proxy for queueing. Swift is part of the lineage that fed into Falcon.

Falcon (Google's hardware reliable transport)

Falcon is Google's answer to "what if we just built a reliable transport in hardware on top of Ethernet?" — a Layer-4 transport offloaded into the NIC, with sub-microsecond loss detection, hardware retransmits, and congestion control that descends from Swift. Google contributed Falcon to the Open Compute Project and is now the substrate underneath Ultra Ethernet on Google's own infrastructure.

For a deeper dive into what Falcon means for cloud GPU economics, see decentralized GPU compute and the discussion of cross-provider fabric variance.

SRD (AWS EFA)

AWS's Scalable Reliable Datagram is the most operationally radical of the bunch: SRD does not provide in-order delivery. Instead it sprays packets across all available paths and lets the upper layer (libfabric, then NCCL/MPI) re-order. This removes head-of-line blocking entirely and makes EFA remarkably resilient to single-link issues, at the cost of needing application-level tolerance for out-of-order delivery — which collectives are fine with.

What this means for picking a fabric

If you operate your own cluster on RoCE, you are configuring DCQCN (or a vendor's DCQCN variant) whether you know it or not. Plan for it: instrument PFC pause durations, ECN mark counts, and per-port queue depths. Meta's SIGCOMM paper is explicit that the operational discipline around congestion control was harder than the protocol choice itself.

If you rent from a hyperscaler, the congestion-control choice is the cloud's problem; what you should measure is outcome tail latency (P99 collective time) across runs. EFA and Falcon are designed to make the protocol choice invisible — verify by benchmarking, not by reading datasheets.

Topology choices: fat-tree, rail-optimized, dragonfly, HPN dual-plane

A topology is a graph of switches and links chosen to make the worst-case bisection bandwidth and tail latency acceptable for your collective workload. Four families dominate AI clusters in 2026.

Fat-tree (Clos)

The classic three-tier datacenter topology: leaf, spine, super-spine, with enough uplinks at every layer to maintain full bisection bandwidth. Fat-tree is well-understood, supports any traffic pattern, and is the default for general-purpose datacenters.

For AI specifically, fat-tree is expensive at frontier scale. To maintain full bisection across 16k GPUs at 400 Gb/s, you need a lot of expensive top-of-fabric switches. The cost-per-bit at the top of the tree dominates.

Rail-optimized

NVIDIA's reference design and the topology behind every modern DGX SuperPOD: instead of one logical network across all 8 NICs per node, you build 8 parallel rail networks. GPU 0 on every node connects to rail 0 only; GPU 1 to rail 1 only; and so on. Each rail can be a much smaller fat-tree because it carries 1/8 the traffic.

This works because NCCL is rail-aware: when 8-way TP runs inside the node on NVLink and 8-way DP runs across nodes, the DP collective for GPU 0's slice never touches rail 1. The result is dramatic switch-count savings without losing collective bandwidth — for the workloads it was designed for.

The downside: it bakes in the assumption of 8-way symmetric parallelism. Mixed workloads, MoE all-to-all that doesn't respect rails, or research codes that don't pin ranks to rails will see uneven utilization.

Dragonfly+ (and dragonfly)

A two-level topology used in HPC systems (Cray Slingshot, parts of Frontier and Aurora supercomputers): a small number of "groups" with rich intra-group connectivity, plus a much smaller number of long links between groups. Dragonfly has shorter network diameter than fat-tree at frontier scale and uses fewer cables — important when cable count and length become physical constraints.

The trade-off is adaptive routing complexity: dragonfly works well only if your switches can spread traffic across group links intelligently, which requires hardware-level adaptive routing (NVIDIA Quantum-2 and Slingshot both do this). For AI clusters that don't have an HPC heritage, dragonfly is rare; for converged HPC-AI sites it's increasingly attractive.

HPN dual-plane (Alibaba)

Alibaba's HPN (Qian et al., SIGCOMM 2024) takes a different cut: build the cluster as two independent Ethernet planes for redundancy and load balancing, then expose both planes to the host. The host picks per-flow which plane to use. The result is failure tolerance (one plane can degrade without halting training) and natural load spreading without ECMP entropy problems.

HPN is now a reference design for very large RoCE clusters. Meta's GenAI cluster paper describes a similar two-plane intuition for fault tolerance, though with different mechanics.

Choosing

Dedicated AI cluster, single workload class, 1k–32k GPUs → rail-optimized.
Mixed AI + HPC, 10k+ accelerators → dragonfly or hybrid.
Very large RoCE-only, hyperscale operator → HPN dual-plane.
Smaller cluster, flexibility valued over cost → fat-tree.

For the parallelism strategies these topologies are sized for, see distributed LLM training, and for the rack-internal companion fabric see NVLink and rack-scale topology.

AI workload-aware routing and adaptive load balancing

The other lever after topology and congestion control is routing — what actually decides which physical link each packet takes.

Why ECMP isn't enough

Classic Ethernet uses ECMP (Equal-Cost Multi-Path), which hashes 5-tuple flow IDs to paths. For AI workloads, this is pathological: NCCL collectives generate a small number of long-lived "elephant" flows, and ECMP's hash collisions cause some links to saturate while others sit idle. The same problem at higher cost shows up on InfiniBand without adaptive routing.

Adaptive routing in InfiniBand and Slingshot

NVIDIA Quantum-2 supports adaptive routing: switches can re-balance flows in hardware based on observed queue depth. Cray Slingshot, HPE's fabric for many of the top supercomputers, ships similar capabilities. For ring all-reduces this is mostly a non-issue; for all-to-all (MoE), bisection-pressuring patterns benefit enormously.

Packet spraying (EFA SRD, Falcon, UEC)

The more radical approach is per-packet spraying: spread each flow's packets across many paths and reassemble at the receiver. AWS's SRD does this, and it is core to Ultra Ethernet's design. The cost is needing receivers that handle reordering; for AI collectives, this is fine because the upper layer is going to barrier anyway.

Topology-aware NCCL

NCCL itself is topology-aware: it builds rings and trees that respect NVLink, PCIe, and IB topology. The NCCL guide covers the env vars (NCCL_TOPO_FILE, NCCL_IB_HCA, NCCL_NET_GDR_LEVEL, NCCL_ALGO, NCCL_PROTO) that let you nudge NCCL toward the right topology when its auto-detection is wrong. A common production fix is making sure NCCL picks the right rail per rank — the wrong choice can cost 30–50% of expected bandwidth without any error.

The bottom line

The collective tail is the real bottleneck in large-scale training. Synchronous all-reduce means the slowest GPU sets the step time, which means the p99 of your network is the metric that drives wall-clock — not the headline bandwidth. Every fabric decision (InfiniBand vs RoCE vs SRD vs Falcon), every topology decision (rail-optimized vs dragonfly vs HPN dual-plane), and every congestion-control decision (DCQCN, HPCC, Falcon's hardware transport) is ultimately about compressing that tail. The single biggest lever is tail discipline: lossless transport, deterministic latency, adaptive routing, and topology-aware collectives.

If you take only this away:

Mean bandwidth is a vanity metric. Optimize p99 collective time or you optimize nothing.
InfiniBand is the default for frontier on-prem; its tail discipline is paid-for, not earned via tuning.
RoCE is competitive at hyperscale (Meta, Alibaba HPN) only if you commit to the operational tax — PFC, ECN, ECMP, and adaptive routing tuning.
Rail-optimized topology beats fat-tree on every collective that matters at >8k GPUs.
NVLink stays inside the rack. The whole inter-rack discussion is about what to do once you've spent its 1.8 TB/s.

For the GPU-side fast fabric this layers on, see NVLink and rack-scale topology. For the collective library that runs across all of it, see the NCCL guide.

FAQ

Q: Should I worry about my network for inference?

Mostly no, unless you're running multi-node inference. Most inference deployments are single-replica (one node). Network is for training and edge cases.

Q: InfiniBand or RoCE?

If you have a choice and a team to operate it: IB for predictable performance. RoCE for cost and operational simplicity.

Q: How much bandwidth do I need?

For a 70B-class model trained on 64 GPUs: 200 Gb/s per port is sufficient. For frontier scale (1000s of GPUs): 400 Gb/s minimum, 800 Gb/s preferred.

Q: Can I mix InfiniBand and Ethernet?

For different purposes, yes. IB for collectives, Ethernet for management/storage. Don't mix on the same logical fabric.

Q: How do I know if my network is the bottleneck?

Profile a training step. If GPU compute time + memory I/O time + NVLink time + IB time = step time, you're network-bound when IB time is large.

A typical healthy ratio: <30% of step time on inter-node communication.

Q: GB200 NVL72 — is it worth it?

For frontier-scale (>10k-equivalent compute) training: yes, if you can afford it. For everything else: standard DGX H100/B200 is enough.

Q: How do I migrate from IB to RoCE (or vice versa)?

Carefully. The hardware change is significant. Validate with nccl-tests at every step. Don't switch a production cluster mid-run.

Q: Will optical/photonic networking change this?

Yes, but slowly. Co-packaged optics (CPO) reduce power and improve density. NVIDIA's Rubin gen is expected to integrate CPO. Expect 2-3 years before mainstream impact.

Q: How is networking for inference servers different?

Mostly single-replica, so no inter-node networking. Network within a replica is NVLink-only for TP. The only inter-node concern: load balancing requests across replicas, which is normal HTTP load balancing.

Q: Should I deploy GB200 NVL72 or 9× standard DGX H100?

For frontier training where TP > 8 is required: GB200.

For most workloads: 9× DGX is more flexible. Different jobs can use 8-GPU islands.

Q: How do I detect tail latency in collectives?

Profile with NVIDIA Nsight. Per-rank timing of each collective. P99 collective time reveals tails.

In production: instrument NCCL with NCCL_DEBUG=INFO and timestamps. Aggregate via Prometheus.

Q: What if my IB switch firmware is old?

Update it. Modern firmware has critical bug fixes for adaptive routing, congestion control, and edge cases. Most clusters update annually.

Q: How do I select between rail-optimized and fat-tree topology?

Rail-optimized is right for AI-specific workloads at scale. Fat-tree is right for mixed workloads or smaller clusters.

For dedicated AI clusters: rail-optimized. For mixed-purpose: fat-tree.

Q: What's an acceptable bandwidth efficiency for nccl-tests?

For NDR IB rail-optimized: 80-90% of theoretical peak.

If you're below 60%: investigate. Misconfiguration somewhere.

If at 60-80%: room for improvement; check NCCL env vars.

If above 90%: well-tuned.

Q: How does this differ for AMD clusters?

RCCL is AMD's NCCL equivalent. API-compatible. Performance characteristics similar but software ecosystem (debugging tools, documentation) lags.

For AMD clusters: same principles, slightly different operational tooling.

Q: What about training across two datacenters?

Hard. Even modern fiber between datacenters has 10ms+ RTT. NCCL collectives can't cope.

For cross-datacenter training: use federated approaches (DiLoCo). Don't try synchronous training across datacenter boundaries.

Q: Will Ethernet (RoCE) replace InfiniBand?

Long-term, plausible. Ethernet's economies of scale are stronger. RoCE gradually maturing.

In 2026, IB still leads on AI-specific performance. RoCE is catching up. By 2028-2030, Ethernet may be on parity.

Q: How important is latency vs bandwidth?

Depends on collective:

Small messages (TP at small batch): latency-dominated.
Large messages (DP at large batch): bandwidth-dominated.
Mixed (most production): both matter.

For most AI training, both are important. Don't sacrifice one for the other.

Q: InfiniBand vs RoCEv2 — what's the actual difference in 2026?

InfiniBand is a purpose-built fabric (link layer, transport, and management plane all custom) optimized for low-latency RDMA. RoCEv2 runs the InfiniBand transport headers over UDP/IP on commodity lossless Ethernet, which lets you reuse Ethernet switching and routing. In 2026 the per-port bandwidth (400/800 Gb/s) is identical, and the achievable collective bandwidth is within 10–20% if both are tuned well. The remaining differences: IB has lower switch-to-switch latency (~1–2 µs vs 5–10 µs), simpler operations on small clusters, but stricter vendor lock-in (NVIDIA Quantum). RoCEv2 is cheaper at scale, integrates with existing Ethernet, but demands disciplined PFC/ECN configuration — see Meta's SIGCOMM 2024 paper for the canonical operational lessons.

Q: What is AWS EFA and how does it compare to InfiniBand?

EFA is AWS's Elastic Fabric Adapter — a custom OS-bypass transport (SRD) that runs over AWS's underlying datacenter Ethernet. Unlike RoCE, SRD does packet spraying instead of requiring in-order delivery, which makes EFA very resilient to single-link congestion. Per-instance bandwidth on P5 (H100) is 3.2 Tb/s; on P6 (B200) it's higher still. Performance is competitive with NDR IB on most NCCL collectives, with somewhat higher tail latency on all-to-all-heavy workloads. EFA is fine for almost all training; it's not as predictable as dedicated IB at >16k-GPU scale.

Q: What is Google Falcon?

Falcon is Google's hardware-offloaded reliable transport over Ethernet, contributed to the Open Compute Project. It targets InfiniBand-class latency and loss recovery without InfiniBand's vendor lock-in, and is part of Google's Ultra Ethernet strategy. Behind the scenes, Falcon descends from earlier RTT-based congestion control (Swift, Timely) and sits underneath GCP A3/A4 GPU instances.

Q: What is Ultra Ethernet and should I wait for it?

The Ultra Ethernet Consortium (AMD, Broadcom, Cisco, Google, HPE, Intel, Meta, Microsoft and 50+ others; see ultraethernet.org) is standardizing an Ethernet-based transport for AI/HPC that competes with InfiniBand on tail latency. UEC v1.0 spec landed in 2024–2025; first compliant silicon and switches show up in 2026. If you are buying a new cluster in 2026 for delivery in 2027, UEC is a credible alternative to IB. If you are buying today, IB and tuned RoCEv2 are the production-ready answers.

Q: 400G vs 800G Ethernet — when does 800G pay back?

For training workloads that are network-bound on DP all-reduce or MoE all-to-all at >1024-GPU scale, the step-time improvement from 400G → 800G is real (1.5–2× on bandwidth-bound phases) and the cost premium is usually 50–80%. For training that is compute-bound or for inference, 400G is more than enough. The bigger question is often whether your switches support 800G end-to-end (Quantum-3 / Spectrum-X 800G / Tomahawk 5) rather than whether the NICs do.

Q: DAC, AOC, or LPO — which optics for an AI cluster?

DAC (direct attach copper) for runs under ~3 m (mostly intra-rack). AOC (active optical cable) for 5–30 m runs where you need flexibility. LPO (linear-drive pluggable optics) is the 2026 story for top-of-rack to spine connectivity: 30–50% lower power than traditional retimed optics, available at 400G and 800G from multiple vendors, no DSP to fail. Co-packaged optics (CPO) is the next step but mostly a 2027+ deployment.

Q: What does Meta's RoCE-at-scale paper actually say?

Three things. First, RoCEv2 can run at 32k-GPU GenAI scale, despite years of "it can't be done at hyperscale" folklore. Second, the work is in operations, not protocol design: rail-optimized topology, careful receiver-side congestion control, PFC headroom math, and observability. Third, the failure modes are different from IB — fewer link-level errors but more subtle congestion-driven tail spikes. Read Gangidi et al., SIGCOMM 2024 directly; it's the most useful single paper on AI cluster networking published in the last five years.

Q: How does cluster networking interact with checkpointing and recovery?

Checkpoint writes go over a separate storage network, not your collective fabric — but you still need bandwidth headroom on the storage side because frontier checkpoints (DeepSeek-V3 671B at FP8 is ~700 GB) take meaningful time to write. See checkpoint storage and recovery for the storage-side math; this guide assumes the collective fabric and storage fabric are sized independently.

Q: Can I run AI training on consumer Ethernet (10/25/40 Gb/s)?

For small jobs (< 16 GPUs, single-node), yes. Just slower.

For frontier-scale training: no. Need at least HDR/NDR IB or equivalent RoCE.

Q: What about 800 Gb/s XDR availability in 2026?

Limited. NVIDIA Quantum-3 switches are shipping but most clusters still use NDR.

By 2027, expect XDR to be more common. Wait if your cluster is being designed now and you can.

Q: How do I size network bandwidth per GPU?

Modern recommendation: at least 1× the GPU's HBM bandwidth, divided by number of GPUs sharing a NIC. For H100 (3 TB/s HBM, 1 NIC per GPU): 400 Gb/s NIC = 50 GB/s. Roughly 1/60 of HBM. That's the standard ratio.

Q: What's the cost of a fully-equipped 1024-GPU AI cluster network?

Typical 2026 numbers:

8 NICs per node × 128 nodes = 1024 NICs at $1000-2000 each = $1-2M.
16-32 leaf switches (depending on radix) at $30k each = $480k-1M.
4-8 spine switches at $80k each = $320k-640k.
Cables: $200k.
Total network: $2-4M.

Plus power, cooling, datacenter space. Networking is a non-trivial fraction of cluster cost.

Q: How does ROCE compare on AWS EFA specifically?

AWS EFA performance on p5.48xlarge: typically 70-80% of dedicated NDR IB at the same nominal bandwidth.

Most workloads see this gap as acceptable. Some collective patterns (all-to-all for MoE) hit harder.

Q: What about Cerebras and other non-NVIDIA training?

Cerebras CS-3 has its own SwarmX interconnect. Different paradigm. Doesn't use NCCL/IB.

For most teams, irrelevant — Cerebras is a separate ecosystem.

Q: How do I monitor the cluster network health during a long training run?

Continuous metrics:

Per-port bandwidth and error counters.
Per-rank collective latency P50/P95/P99.
Tokens-per-second drift over time.

Alert on:

Bandwidth drop >10% from baseline.
Error count increase.
Tail collective time spike.

Q: Should I use NIC bonding (LACP) for redundancy?

For management traffic: yes. For NCCL collectives: no — NCCL handles multi-NIC natively without LACP.

Q: What's the role of the subnet manager (OpenSM)?

OpenSM (or NVIDIA UFM) discovers IB topology, routes traffic. Critical for IB clusters.

Misconfigured subnet manager = subtly broken IB. Run a healthy SM; monitor it.

Q: Can I use 100/200 Gb/s IB for AI training in 2026?

For smaller models (< 70B), yes. Throughput will be limited by network on multi-node training.

For frontier training: 400 Gb/s minimum.

Q: What is SHARP and when does it actually help?

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) is NVIDIA's in-network reduction primitive on InfiniBand switches. The switch ASIC performs the sum/max/min of incoming tensor chunks instead of forwarding them to a designated host for reduction. For all-reduce operations on large dense tensors (DP gradient sync), SHARP cuts the collective latency by 30-50% at large scale by avoiding the bandwidth-doubling of ring-allreduce. It is a meaningful win for >1024-GPU jobs doing FSDP or DDP; for jobs under 256 GPUs the win is smaller and sometimes negative because of fixed setup cost. SHARP requires Quantum-2 / Quantum-3 switches and NCCL ≥ 2.20 with SHARPv3 support. RoCE has no native equivalent — Ultra Ethernet v1 proposes in-network reductions but production hardware is just landing in 2026.

Q: How does DCQCN actually work?

DCQCN combines ECN (Explicit Congestion Notification) at switches with a host-side rate-adjustment algorithm modeled on QCN. Switches mark packets with CE (Congestion Experienced) when their queue exceeds a threshold (typically Kmin/Kmax in WRED). Receivers detect these marks and send CNPs (Congestion Notification Packets) back to senders. Senders multiplicatively decrease their rate on CNP and additively increase it during quiet periods. The DCQCN parameters that matter in production: Kmin=300KB, Kmax=3MB, Pmax=10%, RP timer 55µs, alpha update 50µs. These are NVIDIA's recommended defaults; Meta tuned them lower for 32k-GPU scale. Misconfigured DCQCN is the most common cause of RoCE collective stalls — see the Meta SIGCOMM 2024 paper for the failure mode analysis.

Q: What is HPCC and how does it compare to DCQCN?

HPCC (Li et al., SIGCOMM 2019) is an alternative congestion-control scheme that uses INT (In-band Network Telemetry) to give senders exact per-hop queue state instead of binary ECN marks. Senders adjust to precise link utilization, achieving near-zero queueing at low load and quick convergence under bursts. HPCC outperforms DCQCN on workloads with bursty traffic (incast scenarios, MoE all-to-all). The catch: it requires INT-capable switches (Tofino, some Broadcom Trident lines) and host-side INT processing. Production usage is mostly Alibaba HPN; Meta uses a DCQCN variant with custom tuning rather than HPCC.

Q: What's LPO (linear-drive pluggable optics) and why is everyone talking about it?

LPO removes the DSP from the optical module. A standard 400G QSFP-DD module has a DSP that does CDR, FEC, and equalization — drawing 12-18W per module. LPO modules remove the DSP and rely on the host switch's SerDes to do equivalent processing, dropping module power to 6-10W. At 800G the math gets even better: LPO 800G modules at ~10W vs ~18W for retimed DSP modules. For a 1024-port cluster fabric, switching to LPO can save 8-12 kW just in optics. Caveat: LPO requires tight signal-integrity discipline on the switch ASIC and short cable runs (< 50m typical). Standard now on the latest Quantum-3 and Spectrum-X switches.

Q: When does co-packaged optics (CPO) become real?

CPO integrates the optical engine directly into the switch ASIC package, eliminating pluggable interfaces entirely. NVIDIA announced Quantum-X Photonics CPO switches in early 2025 with 102.4 Tb/s switching capacity. Production deployments begin landing in late 2026 with Rubin platform clusters. The win: another ~30% power reduction over LPO, and the ability to scale switch radix to thousands of ports per chassis. The catch: CPO switches cannot be repaired in the field if an optical engine fails — entire line cards or chassis swap. Hyperscale and frontier deployments only; not relevant for sub-1000-GPU clusters.

Q: How does NVLink fabric extend beyond a single GB200 NVL72 rack?

NVLink fabric within a 72-GPU NVL72 rack is fully connected at 1.8 TB/s/GPU. Beyond one rack, GB200 systems revert to InfiniBand (or RoCE) for inter-rack collectives — 800 Gb/s XDR is the standard 2026 inter-rack fabric. NVIDIA's published Rubin Ultra and successor architectures hint at multi-rack NVLink fabric (NVLink Switch System scaling to 576 or more GPUs), but as of mid-2026 these are roadmap, not shipping. The practical pattern: TP within a rack on NVLink, PP/DP across racks on InfiniBand. See the NVLink and rack-scale topology guide for the in-rack side.

Q: What's the cost of running RoCE wrong?

Concrete failure modes seen in production: (1) PFC misconfiguration causes head-of-line blocking, dropping NCCL bandwidth to 10% of nominal; (2) ECN markers missing, DCQCN can't engage, microbursts collapse all-reduces from 5 µs to 50 ms; (3) MTU mismatch between switches and NICs causes silent packet fragmentation, halving effective throughput; (4) buffer underprovisioning causes incast-driven drops, NCCL retransmits, training step time inflates 2-5x at random intervals. Each of these is recoverable but takes 1-4 weeks of dedicated SRE work to diagnose. Budget 1-2 network engineers full-time for any >256-GPU RoCE deployment.

Q: Is Ultra Ethernet ready to deploy in 2026?

UEC v1.0 specifications shipped in late 2024. First UEC-compliant switches (Broadcom Tomahawk 6, Marvell Teralynx 10) and NICs (AMD Pensando, Broadcom Thor 2) start landing in mid-2026. The honest answer: UEC v1.0 is production-deployable for greenfield clusters in 2026-Q4 with vendor-supported configs. The known caveat: software-side ecosystem (collectives library equivalent to NCCL on Ultra Ethernet) is still maturing. Most 2026 production RoCE deployments are sticking with vendor-specific extensions (Meta's, Alibaba's, AWS's) rather than betting on UEC v1.0 maturity.

Q: How do AWS EFA and SRD differ from InfiniBand?

EFA is the user-space transport API; SRD is the protocol it speaks. SRD does multipath spraying — splits each large message into thousands of small packets and load-balances them across all available paths simultaneously. It tolerates out-of-order packet delivery (the receiver reassembles). This gives EFA much better tail-latency performance than RDMA's single-path reliable connection model under congestion, at the cost of slightly higher base latency (~15-25 µs vs InfiniBand's 1-2 µs). For frontier training at AWS scale (Trainium UltraClusters, P6 H200 clusters), SRD's congestion behavior is the entire reason AWS can run RoCE-equivalent workloads on standard Ethernet without InfiniBand.

Q: What about training across two datacenters separated by 10-100km?

This is where DiLoCo and similar reduced-communication training algorithms become relevant. Inter-DC bandwidth is typically 100-400 Gb/s aggregate per link, with 100-500 µs latency (much higher than intra-DC's 1-10 µs). Synchronous SGD with standard all-reduce becomes impossible at this scale because every step waits hundreds of microseconds. Production 2026 solutions: (1) DiLoCo-style outer-optimizer training with infrequent sync (every 500-5000 steps), (2) pipeline parallelism with one DC per pipeline stage, sized so per-stage compute exceeds inter-DC latency. The decentralized GPU compute guide discusses adjacent bandwidth-reduction techniques.

Q: How do I size network bandwidth per GPU for inference vs training?

Training (FSDP gradient sync, dense): ~50 Gb/s per GPU sustained for 70B-class models, ~150 Gb/s for 400B-class. With 8 GPUs per node and rail-optimized topology, that's 400 Gb/s per node minimum, 1.2 Tb/s for big models. Inference (disaggregated prefill/decode): KV transfer is the dominant traffic, ~5-20 Gb/s sustained per active session. Tensor parallelism for inference is intra-node only (NVLink); inter-node inference traffic is much lighter than training. Most inference clusters can run on 200 Gb/s per node and be fine; training clusters need 400-800 Gb/s.

Q: What happens when a NIC fails mid-training?

In rail-optimized topology, losing one NIC takes that rail offline for the affected node. NCCL's default behavior: hang the collective indefinitely. Production response: (1) job orchestrator detects the failure via NCCL timeout (typically 30-300s), (2) checkpoint and restart the job, swapping the faulty node, (3) resume from the last checkpoint. Modern systems (NeMo, Mosaic, MegatronLM) automate this with elastic launching. Mean time to recovery: 5-15 minutes for a well-tuned system, 1-3 hours for a poorly-tuned one. Hardware NIC failure rate at scale: ~1 per 10,000 GPU-days. For 16k-GPU clusters, expect a NIC failure roughly every 12-18 hours. See checkpoint storage and recovery.

Q: How do I benchmark a new cluster before training on it?

Run nccl-tests in this order: (1) all_reduce_perf -b 8 -e 8G -f 2 -g 8 within one node to verify NVLink baseline (should hit 80%+ of NVLink theoretical); (2) same test across 2 nodes to verify inter-node BW (should hit 80%+ of NIC theoretical); (3) gradually scale to full cluster size. (4) Compare bus bandwidth at 1 GiB message size to vendor's published numbers; anything < 80% indicates configuration issues. (5) Run for 1-4 hours and watch for tail outliers — any single iteration > 2x median is a red flag. Published baselines: 350-400 GB/s sustained busbw for 8-GPU NVLink, 25-40 GB/s sustained busbw for 400 Gb/s inter-node ring all-reduce.

Q: Does FP8 training change network bandwidth requirements?

Yes, materially. FP8 weights and FP8 gradients halve the bytes transferred per all-reduce step. For Llama-3 70B FSDP, gradient sync at FP8 transfers ~70 GB per step vs ~140 GB at BF16. The collective compute on the wire is unchanged in structure but the absolute byte count drops. The net effect: a cluster that was network-bound in BF16 may become compute-bound in FP8, shifting the optimization target. See FP8 training tradeoffs.

Q: How does the Llama-3 405B training cluster compare to xAI's Colossus?

Llama-3 405B was trained on a 24k-GPU H100 cluster with NDR InfiniBand. xAI's Colossus (2024) ran on 100k H100s; Colossus 2 (announced 2025) scales to 200k Blackwell GPUs with XDR InfiniBand. Both use rail-optimized topology with SHARP. The notable difference: Meta increasingly publishes on RoCE-at-scale for serving, while xAI has been consistent with InfiniBand for training. Anthropic and OpenAI cluster details are less public; both are believed to use InfiniBand for frontier training based on hardware-vendor announcements.

Q: What does the network bill look like for a 16,384-GPU cluster?

Rough 2026 numbers, InfiniBand XDR rail-optimized: ~2048 server-side ConnectX-8 NICs ($3-5k each) + ~512 Quantum-3 leaf switches ($80-120k each) + ~128 spine switches ($150-200k each) + ~4-8k optical modules (LPO at $400-800 each) + 16-32 km of fiber. Total fabric capex: $80-150M. Compare to GPU capex at ~$40k × 16384 = $655M. Networking is roughly 12-23% of cluster capex but determines whether the cluster trains a frontier model in 60 days or 90 days — a 50% time-to-train delta justifies premium fabric.

Network design for new clusters

If you're building an AI cluster from scratch in 2026, here's the playbook.

Step 1: define workload

What models do you train? What sizes? What batch sizes? Prefill: decode mix?

This determines the per-job network requirements.

Step 2: pick GPU type and count

Based on workload, pick H100 or B200, count and pricing tier.

This determines per-node compute and memory; networking flows from there.

Step 3: pick topology

For AI-specific: rail-optimized (NVIDIA reference). For mixed-workload datacenters: fat-tree.

Step 4: pick fabric (IB or RoCE)

If on-prem and dedicated: IB. If cloud or mixed: RoCE. If you have IB expertise: IB. If you don't: RoCE may be more practical.

Step 5: size bandwidth

Per-port: 400 Gb/s NDR is the modern minimum. 800 Gb/s XDR if available and budget permits.

Step 6: design switches

Fixed-cap radix-1024 NDR (or equivalent). Ratio of leaf to spine switches based on bisection bandwidth target.

For 1024 GPUs: typical 32 leaf + 8 spine.

Step 7: cabling and physical layout

Plan for cooling. Liquid cooling in dense racks.

Cable management is non-trivial at scale. Allow time and budget.

Step 8: deploy and validate

Run nccl-tests on assembled cluster. Verify achieved bandwidth matches design.

Iterate on configuration until performance meets targets.

Step 9: production monitoring

Set up metrics, alerts, automated diagnostics.

The network is the foundation. Get it right.

Network performance debugging deep dive

When the network underperforms, here's the diagnostic workflow.

Symptom: nccl-tests below expected

Step 1: Verify topology detection.

NCCL_DEBUG=INFO ./build/all_reduce_perf -b 1G -e 1G -f 1 -g 8 2>&1 | grep -i "topo\|tree\|ring"

Wrong topology = NCCL chooses suboptimal path.

Step 2: Check P2P.

nvidia-smi topo -m

PHB instead of NV# = NVLink not used.

Step 3: Check IB status.

ibstatus
ibcheckerrors

LinkUp + zero errors = IB healthy.

Step 4: Verify GDR.

NCCL_DEBUG=INFO ./build/all_reduce_perf 2>&1 | grep "GDR"

Should see "Using GDR" for each rank.

If all four checks pass and bandwidth is still low: deeper investigation (NUMA, kernel, driver).

Symptom: random latency spikes

Step 1: Check for stragglers.

Per-rank step time histograms.
Identify nodes consistently slower.

Step 2: Check PFC pause times (RoCE).

High pause = congestion.
Adjust ECN thresholds.

Step 3: Network buffer overflow.

Check switch port queue depth.
May need larger buffers.

Step 4: CPU jitter affecting NIC interrupts.

Pin processes to specific CPU cores.
Disable C-states.

Symptom: training throughput below baseline

Compare achieved tokens/sec to reference (Llama-paper, NVIDIA-published numbers).

Common gaps:

Network: 10-30% below = network issue.
Compute: 10-20% below = framework issue.
Combined: 30-50% below = misconfiguration.

Profile with NVIDIA Nsight to localize.

Symptom: silent NCCL slowdown

Sometimes NCCL works but at reduced bandwidth.

Causes:

Cosmic-ray bit flips on a NIC.
Driver bug on a specific node.
Switch firmware issue.

Mitigation: monthly nccl-tests baseline. Alert on >10% degradation.

Tools and techniques

NVIDIA Nsight Systems: per-GPU timeline.
NCCL_DEBUG=INFO: NCCL's view.
ibstat / ibstatus: IB-level health.
DCGM: NVIDIA's data center GPU manager.
Custom Prometheus metrics: per-port and per-rank.

For frontier-scale clusters, all of these are essential.

Cost-quality trade-offs

Networking choices have economic implications.

Bandwidth tier vs cost

For a 1024-GPU cluster:

Bandwidth	Cluster network cost	Throughput vs HDR baseline
HDR (200 Gb/s)	$1.5M	1.0×
NDR (400 Gb/s)	$2.5M	1.6×
XDR (800 Gb/s)	$4.5M	2.3×

Cost premium not linear with bandwidth. Diminishing returns at top tier.

For most teams in 2026: NDR is the sweet spot. XDR for frontier deployments where every percent matters.

Topology vs cost

Rail-optimized vs fat-tree for 1024 GPUs:

Topology	Cluster network cost	Workload flexibility
Rail-optimized	$2.5M	AI-only
Fat-tree (full bisection)	$3.8M	Any workload

Fat-tree premium of ~50% buys flexibility. For dedicated AI clusters, rail-optimized wins economically.

Vendor diversity

Single vendor (NVIDIA only): simplest. Single point of failure. Multi-vendor: more resilient, harder to operate.

Most production: single vendor (NVIDIA + Mellanox/NVIDIA networking). Diversification is rare due to operational cost.

Failure modes deep dive

Specific failures that affect production training.

NIC firmware bug

Symptom: random NCCL hangs on specific nodes.

Diagnosis: review IB error counters. Specific firmware bug may be public knowledge.

Fix: update firmware. Some bugs require switch firmware too.

Cable issue

Symptom: link works but at reduced speed.

Diagnosis: check link rate via ibstat. Bad cable = falls back to slower speed.

Fix: replace cable. Use specified cable for distance and speed.

Switch port congestion

Symptom: tail latency spikes during heavy traffic.

Diagnosis: switch buffer fills up. Check counters via switch CLI.

Fix: balance traffic across rails. Or upgrade switch capacity.

NUMA imbalance

Symptom: 1.5-2× variance in per-rank performance.

Diagnosis: process pinning is wrong; some ranks run on far-NUMA cores.

Fix: explicit NUMA pinning. Use numactl.

Driver upgrade regression

Symptom: cluster worked, now slow after driver update.

Diagnosis: new driver has bug. Check vendor errata.

Fix: revert driver. Wait for fix. Test in staging first.

IB subnet manager flap

Symptom: brief interruption, NCCL hangs.

Diagnosis: SM (OpenSM or UFM) crashed and restarted.

Fix: redundant SM. Monitor SM health.

Cooling failure

Symptom: GPU thermal throttling, throughput drops.

Diagnosis: check GPU temperatures. >85C = throttling.

Fix: improve cooling. Check airflow.

Power blip

Symptom: cluster restarts, training resumes from checkpoint.

Diagnosis: facility power issue.

Fix: redundant power. UPS for orderly shutdown.

These all happen in production at scale. Plan for them.

Networking trends and future

Where AI networking is going.

Convergence with HPC

Modern AI networks borrow from HPC (InfiniBand, RDMA, lossless fabric).

The line between HPC and AI cluster networking blurs.

Optical transition

CPO (co-packaged optics) replacing pluggable optics. Power efficiency gains.

Rubin generation (2026-2027) starts integration. By 2028-2029, mainstream.

Higher port speeds

XDR (800 Gb/s) launching now. Expect 1.6 Tb/s by 2027-2028.

Bandwidth scaling continues to outpace compute scaling. Networks stay ahead.

Network-attached compute

In-network reductions (SHARP). Computational logic in switches.

Reduces traffic by performing reductions during transit.

AI-aware fabrics

Networks designed for AI workload patterns. Rail-optimized was the start; further specialization likely.

Software-defined networks

More dynamic control. Adaptive routing, congestion control.

Production deployment is mature; refinement continues.

Photonic switching

Pure-optical switches. Lower latency, higher throughput than electrical.

Research-grade today. Production deployment 5-10 years out.

Frontier cluster networking case studies

Real-world frontier cluster networking.

Meta's Llama-3 cluster

16,000 H100s for Llama-3 405B training.
Rail-optimized topology.
NDR InfiniBand throughout.
8 NICs per node × 2000 nodes = 16,000 NICs.
Bisection bandwidth: ~50 PB/s aggregate.

Engineering challenges:

Tail latency at 16k-rank scale.
Fault tolerance for inevitable failures.
Cross-row collective performance.

Innovations: custom monitoring, automated straggler detection.

NVIDIA's reference DGX SuperPOD

256-1024 H100s in modular configurations.
DGX nodes connected via NDR InfiniBand.
Quantum-2 switches.
Reference designs for various scale tiers.

Available to enterprise customers. NVIDIA supports the full stack.

Microsoft Azure NDmv5 instances

B200 GPUs with InfiniBand interconnect.
NDR speeds.
Available to enterprise customers via Azure.

Performance similar to dedicated DGX. Cloud convenience with frontier capabilities.

CoreWeave clusters

Multi-tenant H100/H200/B200 fleet.
InfiniBand for tenant clusters needing it.
Lower pricing than hyperscalers.

Used by many AI startups for frontier-adjacent work.

What we can learn

Rail-optimized topology dominates.
NDR (400 Gb/s) is the modern standard.
Multi-rail per node (8 NICs).
Aggressive monitoring needed at scale.
Vendor partnerships matter (NVIDIA + InfiniBand).

These patterns are well-established.

Specific networking optimizations for AI training

How AI workloads differ from generic HPC and what that means for networking.

Predictable communication patterns

AI training has predictable patterns: per-step DP all-reduce, per-layer TP all-reduce, etc.

This enables:

Pre-computed optimal routing.
Batched scheduling.
Network-aware framework optimizations.

Bursty bulk transfers

Each step has many small communications followed by computation. Bursty.

Network must handle bursts without dropping or significantly delaying.

Strict synchronization

All ranks must finish each collective before proceeding. Slowest rank dictates.

Tail latency is critical.

Stable working set

Same data flows through same paths repeatedly. Caching and route stability help.

Tolerance for short outages

Most training can recover from brief network outages via checkpointing.

But not from prolonged degradation — performance drops significantly.

Implications for design

Optimize for predicted patterns, not worst-case.
Tail latency more important than mean.
Recovery mechanisms for partial failures.
Stable enough for hours of consistent operation.

Generic HPC networks are over-engineered for some of this; under for others.

Comparing AI cluster networks to other domains

How AI cluster networking differs from other compute domains.

vs traditional HPC

HPC: scientific computing, similar collective patterns, focus on FLOPS. AI: similar networking but different workload mix (more matrix ops, less FFT).

Networking technology overlaps; AI workloads have specific tuning.

vs cloud microservices

Cloud microservices: many small connections, high concurrency. AI: fewer large connections, predictable patterns.

Different design priorities.

vs CDN

CDN: distribute content globally. AI: tightly coupled clusters.

Almost no overlap.

vs HFT (high-frequency trading)

HFT: extreme latency optimization (microseconds). AI: bandwidth optimization, latency matters but less extreme.

Some shared technology (RDMA, kernel bypass).

vs blockchain

Blockchain: many independent nodes, eventual consistency. AI: tightly coordinated, synchronous.

Almost no overlap.

What's unique about AI

Massive bandwidth requirements.
Strict synchronization.
Predictable patterns.
High utilization expected.

These drive AI-specific optimization (rail-optimized topology, NDR/XDR speeds, NCCL specialization).

Cluster commissioning checklist

Steps to bring up a new AI cluster.

Phase 1: Hardware delivery and installation

Verify all hardware against order.
Install per reference architecture.
Cable per topology design.
Power up and BIOS configuration.

Phase 2: Network setup

Subnet manager (OpenSM or UFM) running.
IB/RoCE config per recipe.
Verify port states (LinkUp).
Run ibstat across all hosts.

Phase 3: Software installation

Operating system (typically Ubuntu LTS).
CUDA toolkit.
NCCL.
PyTorch + frameworks.
Container runtime.

Phase 4: Validation

nccl-tests on small subset.
Scale up to full cluster.
Compare achieved bandwidth to design.

Phase 5: Workload bring-up

Small test training run.
Validate per-step time matches expected.
Stress test for stability.

Phase 6: Production rollout

Migrate workloads.
Monitor closely.
Iterate on configuration.

Phase 7: Operational integration

Connect to monitoring.
Define on-call procedures.
Schedule maintenance windows.
Document everything.

This is a multi-week process for a serious cluster. Plan accordingly.

Q: What about 100 Gb/s Ethernet for clusters?

Insufficient for modern training. Use 400 Gb/s minimum.

Glossary

Bisection bandwidth: minimum bandwidth crossing any plane that divides the network in half.
CPO: Co-Packaged Optics. Optical interconnect integrated with chips.
EFA: Elastic Fabric Adapter. AWS's RDMA-over-Ethernet implementation.
HCA: Host Channel Adapter. IB term for NIC.
IB: InfiniBand. NVIDIA's preferred RDMA fabric.
NVLink: NVIDIA's GPU-to-GPU interconnect.
NVSwitch: NVLink fabric switch.
PFC: Priority Flow Control. Lossless Ethernet feature for RoCE.
rail: dedicated network path in rail-optimized topology.
rail-optimized: topology where each GPU position has its own fabric.
RDMA: Remote Direct Memory Access. Bypass CPU on memory transfers.
RoCE: RDMA over Converged Ethernet.

References

Hyperscaler AI-fabric papers

RDMA over Ethernet for Distributed AI Training at Meta Scale — Gangidi et al., SIGCOMM 2024. ACM Digital Library. Meta's account of running RoCE at 32k-GPU scale: rail-optimized topology, custom congestion control, and the operational lessons of "lossless Ethernet" in practice.
Alibaba HPN: A Data Center Network for Large Language Model Training — Qian et al., SIGCOMM 2024. Designing dual-plane Ethernet topologies for trillion-parameter training; the canonical reference for HPN-style fabrics.
Llama 3 Technical Report — Meta, 2024. arXiv:2407.21783. Section on the 16k-H100 training cluster networking, including rail-optimized NDR InfiniBand.

Congestion control and transport

DCQCN: Congestion Control for Large-Scale RDMA Deployments — Zhu et al., SIGCOMM 2015. ACM Digital Library. The canonical ECN-based scheme that makes RoCEv2 viable on lossless Ethernet.
HPCC: High Precision Congestion Control — Li et al., SIGCOMM 2019. In-network telemetry-driven congestion control; the foundation for modern RDMA transports.
Google Falcon — Google Cloud, 2023–2024. Google Cloud Systems Blog. Hardware-offloaded reliable transport targeting InfiniBand-class semantics on standard Ethernet.

Standards and product documentation

InfiniBand Trade Association — infinibandta.org. RoCEv2 specification and IB transport specs.
NVIDIA Quantum-2 InfiniBand platform — nvidia.com/en-us/networking/infiniband/. Quantum / Quantum-2 / Quantum-X switches and NDR/XDR ConnectX HCAs.
NCCL documentation — docs.nvidia.com/deeplearning/nccl/. The collective communications library that sits on top of every IB or RoCE fabric used for training.
NVIDIA DGX H100 networking reference architecture — NVIDIA, 2023. Rail-optimized topology, switch counts, and cabling for 32–4096 GPU pods.

Tail latency and operational fundamentals

The Tail at Scale — Dean & Barroso, CACM 2013. research.google. Why straggler latency dominates large-scale distributed systems — directly applicable to collective-bound training.

Standards and emerging fabrics

Ultra Ethernet Consortium — ultraethernet.org. Industry effort to standardize an Ethernet-based AI/HPC transport competitive with InfiniBand; v1.0 specifications 2024–2025.
AWS Elastic Fabric Adapter (EFA) and Scalable Reliable Datagram (SRD) — aws.amazon.com/hpc/efa/. Reference for SRD's packet-spraying transport.
NVIDIA Spectrum-X and Quantum-X Photonics — nvidia.com/en-us/networking/. 800G Ethernet for AI and the co-packaged-optics roadmap.
Megatron-LM 3D parallelism — Narayanan et al., 2021. arXiv:2104.04473. The canonical mapping of TP / PP / DP onto fabric layers.

Network design for specific workloads

How networking decisions differ by workload.

Pre-training (very large)

Workload characteristics:

All-reduce dominated.
Sustained high bandwidth.
Sensitive to tail latency.

Ideal network:

Rail-optimized or fat-tree.
400-800 Gbps per port.
SHARP-enabled.
1:1 oversubscription.

Cost: highest. Justifies investment.

Pre-training (smaller models)

For models < 100B parameters on 100s of GPUs:

Less stringent requirements.
Standard fat-tree sufficient.
400 Gbps may be enough.

Cost: medium. Standard hyperscaler offering.

Fine-tuning

Less network-intensive:

Mostly data parallel.
Lower aggregate bandwidth needed.
Less sensitive to tail latency.

Cost: low. Cheap networking acceptable.

Inference (TP across nodes)

Critical requirements:

Low latency (every token sees the network).
Consistent performance.
Bandwidth less critical.

Network: usually want single-node. Multi-node inference is hard.

Inference (data parallel)

Less stringent:

Per-replica isolated.
No cross-replica communication.

Cost: low. Standard networking.

MoE training

Heavy all-to-all:

Network dominates many models.
Rail-optimized critical.
All-to-all is harder than all-reduce.

Cost: high. Justify for large MoE.

RLHF / RL

Mixed workload:

Some all-reduce.
Some all-to-all.
Sometimes asynchronous.

Cost: medium-high. Configurable based on specific approach.

Multimodal training

Often heavy data movement:

Image / video data.
Network can be I/O bottleneck.

Cost: medium. Storage network matters too.

Network design framework

For each workload:

Identify dominant operations.
Calculate required bandwidth.
Assess latency sensitivity.
Account for failure recovery.
Design accordingly.

Don't over- or under-engineer.

Networking observability

What to monitor in AI training networks.

Core metrics

For each link:

Bandwidth utilization.
Error rates.
Congestion (queue lengths).
Latency distribution.

For each fabric:

Link state.
Topology consistency.
Congestion hotspots.

Application metrics

Collective operation times.
Per-iteration time variance.
Iteration time vs theoretical.

These map directly to user-visible performance.

Provider-level monitoring

If using cloud:

Per-instance network metrics.
Cross-AZ traffic.
Quota usage.

Tooling

Cluster monitoring (Prometheus, Grafana).
IB-specific tools (perfquery, ibstat).
Application-level (NCCL profiling).
Specialized (RDMA performance tools).

Alerting strategies

Bandwidth utilization > 80%.
Error rates above baseline.
Iteration time variance.
Specific link failures.

Alerts must be actionable.

Distributed tracing

For multi-node debugging:

Trace operations across nodes.
Identify slow paths.
Correlate with network events.

Specialized tooling (e.g., NVIDIA's tools) helps.

Anomaly detection

Patterns to watch:

Hot links.
Cold links (underutilized when expected).
Periodic congestion.
Failure correlation.

Active research area.

What not to monitor

Every individual flow (too much).
Every error (some are expected).

Focus on what affects user-visible performance.

Root cause analysis

When things slow down:

Check network utilization.
Check error rates.
Check application metrics.
Correlate timestamps.
Drill down to root cause.

This is detective work.

Network failure modes

How networks fail in production.

Link failure

A link drops. NCCL operations may hang or slow.

Detection:

Link state monitoring.
Application-level slowdowns.

Recovery:

Routing around failure.
Application restart.

Switch failure

A switch fails. Affects multiple links.

Detection:

Multiple link failures simultaneously.
Region of cluster offline.

Recovery:

Failover to redundant fabric.
Operator intervention.

Cable degradation

Cables fail gracefully — increasing error rates.

Detection:

Error rate monitoring.
Specific link issues.

Recovery:

Replace cable.
Often during scheduled maintenance.

Optical failure

Optics can fail intermittently.

Detection:

Bit error rate monitoring.
Link flapping.

Recovery:

Replace optic.

Software bugs

NCCL, drivers, switch firmware can have bugs.

Detection:

New behavior after upgrade.
Investigation reveals software cause.

Recovery:

Roll back update.
File bug, await fix.

Misconfiguration

Common cause of issues:

QoS misconfigured.
Wrong topology.
Wrong driver versions.

Detection:

Investigation after issues.

Recovery:

Fix configuration.
Document to prevent recurrence.

Capacity issues

When network capacity exceeded:

Congestion.
Packet loss.
Slowdowns.

Recovery:

Reduce workload.
Add capacity.
Better QoS.

Failure isolation

Goal: contain failures to small blast radius.

Patterns:

Redundant paths.
Multiple fabrics.
Failure domains aligned with reality.

Cost vs reliability tradeoff.

Disaster scenarios

Plan for:

Multiple simultaneous failures.
Cascade failures.
Long-lived issues.

Most clusters can handle individual failures. Cascades are harder.

Post-mortem culture

After failures:

Document what happened.
Identify root cause.
Implement preventive measures.

Build institutional knowledge.

Networking outlook for next decade

What to expect in AI networking through 2030+.

Hardware

Per-port bandwidth: 1.6 Tbps becoming standard.
Optical fabrics: emerging.
Co-packaged optics: in deployment 2027+.
New form factors (NVL72-like): proliferating.

Software

NCCL evolution continues.
Open standards (UEC) maturing.
Better debugging / observability.

Topologies

Larger NVLink domains.
Better scaling beyond NVLink boundaries.
Optical interconnect eroding distinctions.

Operations

More automation.
AI-assisted network operations.
Self-healing networks.

Cost

Per-bit decreasing.
Total per-cluster increasing (bigger clusters).
Significant share of total cluster cost.

Trends to watch

AI-specific Ethernet enhancements (UEC).
Photonic interconnects.
New collective operation primitives.
Hardware-software co-design.

What stays the same

Networking remains critical.
Tail latency dominates.
Operational expertise matters.

Implications for practitioners

Stay current. The field evolves fast. Skills atrophy if not updated.

For most: track the field, apply latest where applicable. Build the basics well.

AI networking economics

The economics of AI networking.

Network as cost center

Networks consume:

Capex (switches, cables, optics).
Opex (power, cooling, maintenance).
Engineering investment.

For frontier clusters: 15-30% of total cluster cost.

Network as performance multiplier

Better networking enables:

Larger models.
Faster training.
Higher MFU.

So network spend is performance investment.

ROI calculations

For each $1k spent on networking:

How much MFU improvement?
Translated to training speedup?
Translated to compute cost savings?

This calculation determines optimal network investment.

Cost trends

Per-bandwidth costs decreasing.
Total network cost stable or increasing (more bandwidth needed).
Engineering cost increasing.

Networking continues to be major investment.

Capex vs opex tradeoffs

Building (capex):

Higher upfront.
Lower long-term cost.
Requires expertise.

Buying cloud (opex):

Lower upfront.
Higher long-term cost (typically).
Less expertise needed.

Match to organization stage.

Vendor economics

Networking vendors:

Margins on switching: high.
Margins on optics: high.
Competition on Ethernet (RoCE) vs IB.

This drives industry dynamics.

Networking glossary

Common terms.

NVLink: NVIDIA's GPU-to-GPU interconnect.
NVSwitch: NVLink fabric switch within a node.
NVLink Switch: cross-node NVLink (introduced with GB200).
InfiniBand (IB): high-performance interconnect; preferred by HPC and AI training.
RoCE: RDMA over Converged Ethernet; Ethernet-based RDMA.
RDMA: Remote Direct Memory Access; bypasses CPU for data transfer.
NDR / XDR: IB speeds (400 Gbps / 800 Gbps).
GDR: GPU Direct RDMA; direct GPU-to-NIC DMA.
NCCL: NVIDIA Collective Communications Library.
SHARP: Scalable Hierarchical Aggregation and Reduction Protocol; in-network reductions.
Bisection bandwidth: bandwidth across cluster cut in half.
Fat-tree: classic data center topology.
Rail-optimized: each GPU has dedicated path; reduces contention.
Dragonfly: scale-out topology for very large clusters.
PFC: Priority Flow Control; lossless Ethernet.
DCQCN: Data Center Quantized Congestion Notification; RoCE congestion control.
ECN: Explicit Congestion Notification.
MTU: Maximum Transmission Unit; matters for jumbo frames in RoCE.
Lossless: network without packet drops; required for RDMA.
Tail latency: high-percentile latency; dominates collective operations.
All-reduce: collective op that sums values across all participants.
All-gather: collective op that distributes values from each.
All-to-all: collective op where each pair exchanges.

Networking summary and recommendations

Putting it all together.

Bottom line

Networking is a critical determinant of AI training performance at scale. Insufficient networking caps your effective performance. Excellent networking enables frontier work.

For most teams

You don't build networks — you choose providers. So:

Validate provider's networking matches workload.
Test before committing.
Have fallback options.

For teams building infrastructure

Hire networking expertise.
Plan for scale.
Invest in monitoring.
Document everything.

Hardware recommendations

Today's defaults:

8x H100 / B200 nodes.
IB or RoCE inter-node.
Rail-optimized for >256 nodes.
SHARP for collective acceleration.

Software recommendations

NCCL latest stable.
Topology files where helpful.
Application-level overlap of compute and comms.
Continuous benchmarking.

Cost recommendations

Match networking to workload (don't over-buy).
Plan for failures.
Account for total ownership cost.

Operations recommendations

Monitor everything.
Document procedures.
Train multiple team members.
Plan for incidents.

Future-proofing

Track UEC and other emerging standards.
Plan for next-gen GPUs.
Stay current on NCCL evolution.
Engage with community.

Final advice

Networks make the difference between successful AI infrastructure and dragging effort. Invest accordingly.

This is one of the most leveraged areas of AI infrastructure work.

Networking case studies extended

More case studies of networking design decisions.

Case: Anthropic's training infrastructure

Public details limited. Inferred:

Mix of cloud and dedicated infrastructure.
High-bandwidth networks.
Significant operational investment.

Lessons: top labs invest heavily in networking.

Case: Meta's Llama-3 cluster

Documented:

16k H100s.
Custom RoCE implementation.
Significant networking expertise required.

Lessons: scale + expertise enables custom solutions.

Case: Microsoft / OpenAI

Reportedly:

Massive scale.
Custom network designs.
Multi-DC training capability.

Lessons: hyperscale enables architectural choices unavailable to others.

Case: xAI Colossus

100k+ GPU cluster:

Liquid cooling.
High-bandwidth fabric.
Built rapidly.

Lessons: speed-to-deploy is differentiator.

Case: Smaller labs

Many ML teams use:

Cloud GPU instances.
Standard provider networking.
Optimized application code.

Lessons: don't always need custom infrastructure.

Case: Government clusters

Examples like Aurora, Frontier:

Different networking (Slingshot, etc.).
HPC heritage.
Now adapting to AI.

Lessons: HPC has existing fabric expertise applicable to AI.

Case: Edge / inference clusters

Inference deployments:

Often single-node sufficient.
Standard cloud networking.
Cost-optimized.

Lessons: not every workload needs frontier networking.

Case: Multi-region inference

Some companies serve from multiple regions:

Lower latency to users.
Cross-region for failover.
Routing complexity.

Lessons: networking matters even for non-training workloads.

Case: Hybrid cloud-on-prem

Hybrid deployments:

On-prem for cost-sensitive baseline.
Cloud for burst.
Networking bridge.

Lessons: hybrid networking is its own challenge.

Lessons across cases

Networking matters proportionally with scale.
Expertise is rare and valuable.
Custom solutions for largest deployments.
Standard solutions adequate for most.

Networking specifications by GPU generation

Detailed networking specifications across GPU generations.

A100 era (2020-2022)

NVLink: 600 GB/s per GPU.
NVSwitch: 4.8 TB/s aggregated.
IB: 200 Gbps NDR.
Typical cluster size: hundreds to low thousands.

H100 era (2022-2024)

NVLink: 900 GB/s per GPU.
NVSwitch: 7.2 TB/s aggregated.
IB: 400 Gbps NDR.
Cluster size: thousands to tens of thousands.

H200 era (2024-2025)

Same networking as H100. Memory upgrade only.

B100/B200 (Blackwell, 2024-2026)

NVLink-5: 1.8 TB/s per GPU.
NVSwitch: 14.4 TB/s aggregated.
IB: 800 Gbps XDR (emerging).
GB200 NVL72: 72 GPUs in single NVLink domain.

GB200 NVL72 details

72 GPUs interconnected via NVLink.
130 TB/s aggregated NVLink bandwidth.
Acts as single super-GPU.
Game-changer for very large model training.

Future generations (Rubin, etc.)

NVIDIA roadmap suggests:

Even higher per-GPU bandwidth.
Larger NVLink domains.
Co-packaged optics.

Each generation: significant networking advancement.

Cross-generation transitions

Migrating between generations:

Often requires fabric redesign.
Cabling/optics changes.
Software updates.

Not just buying new GPUs.

Bandwidth scaling laws

Empirically:

Per-GPU bandwidth doubles every 2-3 years.
Cluster size limits scale with this.
Cost per byte transferred decreasing.

Hardware-software co-design accelerates.

AI networking ecosystem

The broader ecosystem.

Component vendors

NVIDIA / Mellanox: IB, NVLink.
Cisco / Arista / Juniper: Ethernet switching.
Broadcom / Marvell: silicon.
Intel: Gaudi networking.

Cloud providers

AWS: EFA + Nitro.
GCP: TPU networking + Compute Engine.
Azure: AKS + GPU pools.
Oracle: H100 / H200 / B200 with networking.
CoreWeave / Lambda: GPU specialists.

Hardware vendors

Supermicro, Dell, HP: server platforms.
ZT Systems: ML platforms.
NVIDIA HGX / DGX reference designs.

Software ecosystem

NCCL (NVIDIA).
RCCL (AMD).
oneCCL (Intel).
Open standards (UCX, UCC).

Standards bodies

IBTA (InfiniBand Trade Association).
IEEE (Ethernet standards).
UEC (Ultra Ethernet).
OCP (Open Compute Project).

Industry trends

Convergence of Ethernet + IB feature sets.
Open standards gaining traction.
Disaggregated networking.
Co-packaged optics.

What's emerging

Massive scale (>1M GPU clusters planned).
Optical fabrics.
AI-specific networking protocols.
New topology designs.

Investment trends

Massive capex on networking:

AI infrastructure accounts for significant portion.
Network now major cost component (10-30% of GPU cluster cost).

This drives innovation.

Consolidation

Networking vendors:

Some consolidation (acquisitions).
New entrants (chip startups).
Reshape over years.

Networking FAQ extension

Common questions and answers.

Q: What's the difference between InfiniBand and RoCE? IB is a complete protocol stack designed for HPC. RoCE is RDMA over Ethernet. Different designs, similar purpose.

Q: Should I use IB or RoCE? For new builds: depends on team expertise, vendor relationships, cost. Both viable.

Q: What is rail-optimized topology? Each GPU connects to a dedicated rail (network plane). Reduces cross-GPU contention.

Q: Why does tail latency matter so much? Collective operations are synchronizing — one slow node blocks all. So tail = total time.

Q: How fast is NVLink vs IB? NVLink: ~900 GB/s per GPU (Hopper). IB NDR: ~50 GB/s per GPU. NVLink is ~18x faster.

Q: Can I use Ethernet without RoCE? Yes, but performance much lower. RoCE / RDMA matters for AI training.

Q: What is SHARP? NVIDIA's in-network reduction. Performs collective ops in switches.

Q: How important is bisection bandwidth? Very. Determines worst-case all-to-all performance.

Q: What about fabric scale limits? Single fabric typically scales to thousands of nodes. Larger requires hierarchical design.

Q: How do I diagnose tail latency? Profile per-node times. Identify outliers. Investigate root cause.

Q: Is IB hardware expensive? Yes, but performance per dollar can be competitive at scale.

Q: Can I run AI training on Ethernet? Yes — RoCE makes it viable. Many large clusters use Ethernet.

Q: How does optical networking fit in? Optical fabrics in development. Could enable bigger clusters with lower latency.

Q: What's UEC (Ultra Ethernet Consortium)? Industry effort to standardize Ethernet for AI workloads.

Q: When will UEC matter? Production deployments emerging in 2026-2027.

Q: How do hyperscalers approach this? Each builds custom networks. AWS EFA, Google's networking, Azure's networking — all different.

Q: Can I bring my own network to cloud? Limited options. Mostly stuck with provider's networking.

Q: How do I size a network? Calculate aggregate bandwidth needed for collective ops. Add headroom for failures and growth.

Q: What's the future of NVLink? Continued evolution. NVLink-5, NVLink Switch fabric for cross-rack NVLink.

Q: How does GB200 NVL72 networking work? 72 GPUs in one NVLink domain. Acts like a single super-GPU. Game-changer for some workloads.

Network operations playbook

How to operate AI training networks.

Daily operations

Monitor link health.
Check utilization patterns.
Investigate any anomalies.
Validate configuration.

Weekly operations

Review aggregate metrics.
Plan capacity.
Update topology if needed.

Monthly operations

Performance benchmarks.
Configuration audit.
Vendor relationship management.

Per-deployment operations

Before each major job:

Validate cluster health.
Run nccl-tests baseline.
Check for any pending issues.

After:

Review performance.
Document any issues.
Update procedures.

Incident response

For network incidents:

Detect via monitoring.
Triage severity.
Investigate root cause.
Mitigate.
Resolve.
Post-mortem.

Standard SRE practice.

Capacity planning

For growth:

Project workload growth.
Identify bottlenecks.
Plan upgrades.
Budget appropriately.

This is a 6-12 month exercise.

Vendor management

For network components:

Maintain relationships with multiple vendors.
Plan procurement timelines (long lead times for some components).
Negotiate support agreements.
Track vendor roadmaps.

Documentation

Comprehensive documentation:

Topology diagrams.
Configuration files.
Operational procedures.
Incident history.

Crucial for team operations.

Training

Network expertise is rare:

Train multiple team members.
Build cross-team understanding.
Document know-how.

This is institutional knowledge.

Continuous improvement

Networks evolve. Continuous improvement:

Monitor industry trends.
Test new technologies.
Iterate on configuration.

Standing still is regressing.

Networking benchmarks and analysis

Concrete benchmark numbers and analysis frameworks.

Bandwidth benchmarks

For 8x H100 SuperPod (single node):

NVLink bandwidth (per GPU): 900 GB/s.
NVSwitch aggregated: 7.2 TB/s.
Bandwidth utilization at 8-GPU all-reduce: ~85%.

For 64-node H100 cluster (InfiniBand NDR):

Per-GPU IB: 400 Gbps = 50 GB/s.
Aggregated bisection: depends on topology.
All-reduce bandwidth: ~30 GB/s typical (Ring algorithm).

These are best-case. Production typically 70-85% of these.

Latency benchmarks

GPU-to-GPU intra-node (NVLink):

5-10 microseconds for small messages.

GPU-to-GPU inter-node (IB):

5-10 microseconds (one-hop).
10-25 microseconds (multi-hop fat-tree).

Latency variance is the bigger issue for AI training.

MFU impact analysis

Model Flop Utilization (MFU) for typical configurations:

Cluster	Network	MFU
Single node	NVLink only	50-60%
16 nodes	IB NDR	40-50%
256 nodes	IB NDR + SHARP	35-45%
1024+ nodes	IB NDR + SHARP, optimized	30-40%

Larger clusters: more communication, lower MFU. Better networks: higher MFU.

Cost-benefit framework

For each network upgrade:

Cost (capex + opex).
Benefit (MFU improvement, throughput increase).
ROI calculation.

Example: upgrading from 200 Gbps to 400 Gbps costs ~$50/GPU/year more, can improve MFU 5-10%, ROI typically positive for large clusters.

Topology comparison

Topology	Cost	Performance	Complexity
Tree	Low	Lower	Low
Fat-tree	Medium	Higher	Medium
Rail-optimized	High	Highest	High
Dragonfly+	Medium-High	High	High

Choose based on scale and budget.

Analysis methodology

For your cluster:

Measure baseline performance.
Identify bottlenecks via profiling.
Quantify potential gains.
Compare to upgrade costs.
Decide.

This is the framework for network investment decisions.

Network design tradeoffs

The fundamental tradeoffs in network design.

Bandwidth vs latency

More bandwidth ≠ lower latency. Different design choices favor each.

For all-reduce: bandwidth dominates. For all-to-all: bandwidth and latency both matter. For sparse models: latency more critical.

Cost vs performance

Higher-performance networks cost more:

Better optics.
More links.
Larger switches.

The cost grows non-linearly with performance.

Reliability vs cost

More reliable networks have:

Redundancy.
Higher-quality components.
Better monitoring.

Cost: 20-50% premium.

Flexibility vs efficiency

Flexible networks (handle many workloads):

More general.
Less efficient for any specific workload.

Specialized networks:

Highly tuned for one workload.
Less flexible.

Build vs buy

Building your own:

Highest control.
Highest cost (and time).
Risk of mistakes.

Buying (cloud):

Faster to start.
Less control.
Predictable cost.

Vendor lock-in

InfiniBand: NVIDIA-dominated. RoCE: more vendor diversity. NVLink: NVIDIA-only.

Lock-in matters for long-term cost.

Standardization vs proprietary

Standard fabrics (Ethernet, IB):

Interoperability.
More options.

Proprietary fabrics:

Optimized for specific use case.
Vendor lock-in.

Power vs performance

Higher-bandwidth fabrics consume more power:

Optical transceivers.
More switches.
Larger cooling.

For scale: power becomes major cost.

Physical considerations

Cable lengths.
Cooling.
Power density.
Building constraints.

These constrain what's possible.

Decision framework

For each tradeoff, weight based on:

Workload priorities.
Budget.
Team capabilities.
Future plans.

There's no universal answer.

Networking for inference vs training

How requirements differ.

Training requirements

High aggregate bandwidth (all-reduce).
Tail latency matters (stragglers).
Failure handling.
Predictable performance.

Optimization: maximize throughput at scale.

Inference requirements

Low latency (per-request).
Consistent performance (p99).
Variable load.
Often single-node or small TP groups.

Optimization: minimize latency, handle bursts.

Hardware differences

Training:

High-bandwidth multi-node fabric.
Many GPUs per cluster.

Inference:

Often single-node sufficient.
Larger fleet of small clusters.

Software differences

Training:

NCCL collectives.
Long-running jobs.
Checkpointing.

Inference:

Per-request inference.
Streaming responses.
Rapid scaling.

Cost differences

Training: amortizes networking cost over long runs. Inference: networking cost is per-request.

For inference: simpler networking often more cost-effective.

Failure modes

Training:

Slow nodes (stragglers).
Network issues across cluster.
Checkpoint corruption.

Inference:

Single-instance failures.
Cascading failures from load.

Operational differences

Training:

Less frequent deployment.
Long debugging cycles.
Cluster-wide changes.

Inference:

Frequent deployment.
Faster iteration.
Per-instance management.

Mixed deployments

Some teams co-locate training and inference:

Shared infrastructure.
Cost optimization.
Operational complexity.

Most: separate.

Network design implications

For training: design for peak performance at scale. For inference: design for cost-efficient single/multi-node serving.

These often lead to different network designs.

Networking strategy for builders

How builders should think about networking decisions.

When you have full control

Building your own DC:

Choose IB or RoCE based on team expertise.
Invest in monitoring.
Plan for scale.

This is the highest-cost, highest-control option.

When using cloud

You inherit cloud's network design:

AWS: EFA + Nitro.
GCP: TPU Pods or Compute Engine.
Azure: GPU pools with various networking.

Optimize within constraints.

Hybrid

Some on-prem, some cloud:

Different network characteristics.
Different operational models.
Plan for the differences.

Decentralized

Variable network quality:

Test each provider.
Lower expectations.
Plan for variance.

Selection criteria

For new deployments:

What's the workload?
What's the scale?
What's the budget?
What's the team capability?

These determine the answer.

Common mistakes

Underinvesting in networking (compute alone doesn't help).
Overinvesting (spending on unnecessary capability).
Mismatched components (high-end GPUs with low-end network).
Ignoring operational complexity.

Successful patterns

Match networking to workload.
Invest in monitoring.
Build operational expertise.
Plan for failure.

These are universal across deployments.

NCCL collective algorithms in depth (ring, tree, NVLS, SHARP)

NCCL is the collective-communication library that sits under PyTorch, JAX, and Megatron, and the choice of algorithm directly determines what fraction of the wire bandwidth turns into useful work. For the operator-facing tuning reference, see the NCCL guide; below is the underlying algorithmic structure.

Ring all-reduce

The textbook algorithm. With N ranks, each rank sends and receives N-1 chunks during the reduce-scatter phase and another N-1 during all-gather. Aggregate per-rank bytes transferred: 2(N-1)/N × message-size, asymptotically 2× the message. Wire efficiency: high for small N, drops as N grows because each chunk traverses one hop per ring step and serial dependencies dominate. Ring is the default for cross-node all-reduce in most NCCL configurations because it is bandwidth-optimal in the steady state. Best when bandwidth is the bottleneck.

Tree all-reduce

A reduction tree (chunks aggregate up the tree) followed by a broadcast tree (results flow back down). Each rank does roughly log(N) hops instead of N. Total bytes per rank scale better with N. Latency improves dramatically for small messages. NCCL uses a double-tree to balance up- and down-traffic. Best when latency dominates: small messages, very large N, or wide trees over many switches.

Hierarchical algorithms (intra-node + inter-node)

For multi-node clusters NCCL composes intra-node algorithms (NVLink-based ring or NVLS) with inter-node algorithms (IB or RoCE ring/tree). Intra-node reduce-scatter, inter-node all-reduce across one rank per node, intra-node all-gather. This pattern minimizes traffic on the slow (inter-node) fabric. Configuration via NCCL_ALGO=Ring,Tree plus NCCL_PROTO=Simple,LL,LL128 and topology hints.

NVLS (NVLink SHARP)

On NVL72-class hardware, NVLink SHARP performs in-network reductions across NVSwitch. Each NVSwitch ASIC sees in-flight gradient packets and reduces them on the silicon, halving the data each NVLink rank has to send back. For all-reduce inside a 72-GPU domain, NVLS provides roughly 2× speedup over a ring on the same NVLink fabric. Enabled by default on GB200 NVL72 racks when NCCL_NVLS_ENABLE=1 and topology is detected.

SHARP (InfiniBand-based in-network reduction)

NVIDIA's Mellanox SHARPv1 and SHARPv2 perform reductions inside Quantum InfiniBand switches. The switch ASIC aggregates contributions from all ports, computes the reduction, and broadcasts the result. For all-reduce, SHARP achieves near-2× the effective bandwidth because each rank only sends its contribution once instead of participating in N-1 hop ring. SHARPv3 on Quantum-3 extends this to larger trees and FP8 reductions. Critically, SHARP is InfiniBand-only — RoCE has no equivalent in production, although the Ultra Ethernet Consortium roadmap includes in-network compute primitives.

Algorithmic bandwidth ceilings

Algorithm	Steady-state bytes per rank	Latency scaling	Best for
Ring all-reduce	~2× message	O(N)	Large messages, moderate N
Tree all-reduce	~2× message	O(log N)	Small messages, very large N
Hierarchical	depends on tiers	Sum across tiers	Multi-node with intra fast
NVLS	~1× message	O(log N)	Within NVL72 / NVSwitch
SHARP	~1× message	O(log N)	InfiniBand fabrics

The headline: SHARP and NVLS halve the wire bytes, which is exactly why InfiniBand + NVLink + SHARP is the frontier-training pattern. RoCE can match raw bandwidth but not the in-network reduction savings.

Per-switch deep dive: Quantum-2/3, Spectrum-X, Tomahawk-4/5, Silicon One

Switch silicon is where the per-port bandwidth, in-network compute features, and congestion-control hooks live. The market splits into InfiniBand (NVIDIA Quantum), AI-tuned Ethernet (NVIDIA Spectrum-X, Cisco Silicon One AI), and merchant Ethernet (Broadcom Tomahawk, Arista using Tomahawk/Trident).

NVIDIA Quantum-2 (NDR 400G)

Quantum-2 is the dominant frontier-training switch as of mid-2026. 64 ports of 400G InfiniBand per switch, 51.2 Tb/s total bandwidth. Supports SHARPv2 in-network reductions, adaptive routing, congestion control. Deployed at xAI Colossus, Meta Grand Teton clusters, Microsoft ND H100, and most large training clusters.

NVIDIA Quantum-3 (XDR 800G)

Quantum-3 doubles per-port to 800G with SHARPv3. Available since 2024-Q4 in volume. The first 800G deployments are GB200 NVL72-anchored clusters; Quantum-3 is the standard scale-out fabric for Blackwell training. Per-switch radix unchanged at 64 ports (51.2 → 102.4 Tb/s aggregate).

NVIDIA Spectrum-X (Ethernet for AI)

Spectrum-X is NVIDIA's Ethernet-side answer for customers who want RoCE rather than IB. Pairs Spectrum-4 ASIC switches with BlueField-3 DPUs and ConnectX-7/8 NICs to deliver adaptive routing, telemetry-based congestion control, and PFC-less operation on Ethernet. Per-port speeds: 400G, 800G. Used at deployments wanting Ethernet for unified networking with Spectrum-X behavior tuned for AI workloads.

Broadcom Tomahawk-4 / Tomahawk-5

Tomahawk-4 (12.8 Tb/s, 25.6 Tb/s variants) and Tomahawk-5 (51.2 Tb/s, 800G ports) are the merchant-silicon foundation for hyperscaler Ethernet AI fabrics. Used by AWS, Meta (for non-Quantum deployments), and various OEMs. Tomahawk-5 supports advanced telemetry, adaptive routing, and the latest 800G Ethernet. The 102.4 Tb/s Tomahawk-6 (1.6T-class) is on the roadmap for 2026-2027.

Cisco Silicon One AI

Cisco's Silicon One platform targets AI-fabric Ethernet. Used by Microsoft Azure and several hyperscalers as an alternative to Tomahawk. Specific AI-focused SKUs add adaptive routing and telemetry tuned for collective communication.

Arista 7800R3 / 7060X

Arista uses merchant Broadcom silicon (Tomahawk-4/5) with their own EOS software. Strong in cloud and enterprise AI deployments; many CoreWeave, Lambda, and second-tier hyperscaler clusters run Arista.

Switch comparison

Switch	Vendor	Aggregate bandwidth	Per-port speed	In-network compute	Production AI use
Quantum-2	NVIDIA	51.2 Tb/s	400G IB	SHARPv2	Dominant frontier-training switch
Quantum-3	NVIDIA	102.4 Tb/s	800G IB	SHARPv3	New 2024-2026 frontier deployments
Spectrum-4	NVIDIA	51.2 Tb/s	400G/800G Eth	Telemetry-driven adaptive	Spectrum-X RoCE clusters
Tomahawk-5	Broadcom	51.2 Tb/s	400G/800G Eth	Adaptive routing	Hyperscaler Ethernet AI
Silicon One AI	Cisco	51.2 Tb/s	400G/800G Eth	AI-tuned features	Azure, select hyperscalers
Arista 7800R3	Arista (Tomahawk)	25.6-51.2 Tb/s	400G Eth	Per-vendor	Cloud + enterprise AI

Selection heuristics

InfiniBand (Quantum-2/3) for frontier training where SHARP and tight latency tails matter. Spectrum-X for Ethernet-preferred deployments wanting NVIDIA's tuned stack. Tomahawk-5 + custom SW for hyperscalers that have the engineering budget to tune RoCE themselves. Cisco / Arista for enterprises that already have those vendor relationships.

Per-NIC deep dive: ConnectX-7/8, Cornelis, Pensando, EFA, gVNIC

The NIC is where RDMA semantics live. For frontier training the NIC choice determines per-server bandwidth, latency floor, and whether SHARP/NVLS-equivalent offloads work.

NVIDIA ConnectX-7

400G per port (NDR IB or 400G Ethernet). Dominant frontier-training NIC. Pairs with Quantum-2 switches for InfiniBand or Spectrum-4 for Ethernet. Supports GPUDirect RDMA, ATS, SR-IOV, hardware-offloaded congestion control. Typical deployment: 8 NICs per H100 server (one per GPU rail).

NVIDIA ConnectX-8

800G per port; pairs with Quantum-3. New for Blackwell deployments. Adds improved congestion-control telemetry, FP8 SHARP support.

NVIDIA BlueField-3 DPU

Combines a ConnectX-class NIC with ARM cores for offloaded networking, storage, and security functions. Used in Spectrum-X deployments as the congestion-control endpoint; the BlueField runs the per-flow telemetry and rate-control logic.

Cornelis Networks (Omni-Path)

Cornelis develops Omni-Path Express, a successor to Intel's discontinued Omni-Path. Targets HPC and emerging AI deployments. Production AI uptake is limited compared to InfiniBand but exists in specialized deployments.

AMD Pensando

AMD's DPU line. Targets cloud and edge networking with optional AI use. Less common in frontier training; more visible in cloud inference.

AWS EFA (Elastic Fabric Adapter)

AWS's proprietary NIC for HPC and AI. Implements SRD (Scalable Reliable Datagram) — connectionless, unordered, packet-sprayed RDMA-like semantics. Bandwidth: 400G per EC2 P5/P5e instance. Tightly integrated with NCCL via the AWS-OFI-NCCL plugin. Used at scale on AWS UltraClusters.

Google gVNIC and Falcon

Google's gVNIC is the virtual NIC for GCE; Falcon is the underlying hardware transport with reliability and congestion control implemented on-chip. Falcon-1 deployed broadly; Falcon-2 announced. Used in GCP A3 / A3 Mega / A4 instances for AI training.

Microsoft Frontier Edge

Microsoft's internal RoCE-style NIC stack for Azure ND H100/H200. Pairs with custom Spectrum-X / Tomahawk deployments. Production at hundreds-of-thousands-of-GPU scale.

NIC comparison

NIC	Per-port speed	Reliability model	Production AI use
ConnectX-7	400G	RC + RDMA (RDMA Reliable Connection)	Frontier IB/RoCE training
ConnectX-8	800G	RC + improved telemetry	Blackwell-era frontier training
BlueField-3 DPU	400G	RC + DPU-offloaded	Spectrum-X RoCE clusters
AWS EFA (SRD)	400G	Connectionless, sprayed	AWS UltraClusters
GCP Falcon	varies	Hardware reliable + sprayed	GCP A3+
Cornelis Omni-Path	400G	Reliable	Niche HPC + AI

Why the NIC matters more than the switch in some setups

The NIC implements the actual reliable-transport semantics, congestion control, GPUDirect RDMA, and SHARP-client logic. A "fast switch + slow NIC" deployment under-uses the fabric; "slow switch + fast NIC" is bottlenecked at the switch. For frontier training, vendor-matched NIC+switch (ConnectX-7 + Quantum-2, ConnectX-8 + Quantum-3, BlueField-3 + Spectrum-4) is the standard pattern.

Ultra Ethernet Consortium v1.0 and the 2026 timeline

The Ultra Ethernet Consortium (UEC) was formed in 2023 by AMD, Broadcom, Cisco, Eviden, HPE, Intel, Meta, Microsoft, and others to define an Ethernet-based AI fabric standard that addresses the gaps RoCEv2 has at scale. UEC v1.0 specifications began rolling out in 2024-2025; production deployments are emerging in 2026.

What UEC v1.0 specifies

Reliable Unordered Delivery (RUD). Decouples reliability from ordering. Packets can arrive out of order and be reassembled at the receiver, eliminating head-of-line blocking. Similar in spirit to AWS SRD and Google Falcon.
Packet trimming. When buffers overflow, switches trim packet payloads but preserve headers, allowing the receiver to detect loss and trigger a fast retransmit without dropping reliability semantics.
Semantic acks. Per-message acks instead of per-packet, reducing ack-storm overhead at scale.
Multi-path congestion control. Connection-level state across many paths, with telemetry from switches feeding back to senders.
In-network compute hooks. Reserved fields for future in-network reduction (analog of SHARP).
Security primitives. Integrated encryption and authentication at link level.

Adoption timeline

2024-Q4: First UEC-compliant switches announced (Broadcom Tomahawk-5 UEC-mode, Cisco Silicon One UEC variants).
2025: Early adopter deployments at hyperscalers (Meta, Microsoft).
2026 (current): Production deployments in select frontier clusters; broader market availability.
2027-2028: Expected to compete with InfiniBand for new frontier training installations.

UEC vs InfiniBand vs RoCEv2

Property	InfiniBand (Quantum-3)	RoCEv2 (untuned)	UEC v1.0
Reliability	Credit-based, lossless	PFC-required	Reliable Unordered Delivery
Ordering	In-order	In-order	Unordered (RUD)
Congestion control	Hardware, adaptive	DCQCN + PFC	Multi-path, telemetry
In-network compute	SHARP	None	Reserved (future)
Vendor lock	NVIDIA-only	Open	Open standard
Production at >32k GPUs	Yes (xAI, Meta, OpenAI)	Yes (Meta SIGCOMM 2024)	Emerging
Operational complexity	Lower	Higher	Medium (new)

Why UEC matters for the IB-vs-RoCE debate

UEC is the credible Ethernet path to "InfiniBand-class" guarantees without the NVIDIA vendor lock. If UEC v1.0 ships on schedule and delivers tail-latency parity with InfiniBand, the long-term winner of the fabric war is likely "Ethernet variants of all kinds" rather than InfiniBand. For 2026, InfiniBand remains the default for new frontier training; UEC is the credible second choice.

DCQCN vs HPCC vs PFC tuning at frontier scale

For RoCE deployments, congestion control is the difference between functional and broken. Two approaches dominate; one is necessary.

PFC (Priority Flow Control)

Link-level pause frames. Switch buffers fill, switch sends PAUSE to upstream sender, upstream stops sending on that priority class. Lossless guarantee. The problem: PFC pause storms. If pauses cascade backward through the fabric, you get head-of-line blocking that locks the entire cluster. PFC must be paired with end-to-end congestion control (DCQCN or HPCC) to avoid this.

DCQCN (Data Center Quantized Congestion Notification)

Microsoft's algorithm. ECN-based: switches mark packets when buffers cross a threshold, senders react by reducing rate, slowly probe up. Tuning parameters: K_min and K_max (ECN marking thresholds), g (rate-decrease aggressiveness), R_AI (rate-increase additive), R_HAI (hyper-additive increase). Default values from RDMA NIC vendors are conservative; tuning for large clusters is a routine but non-trivial exercise.

HPCC (High Precision Congestion Control)

Alibaba's algorithm. Uses INT (In-band Network Telemetry) to compute exact queue lengths and link utilizations, allowing precise rate updates. More accurate than DCQCN's ECN-marking but requires INT support in switches and per-packet metadata bandwidth. Used at Alibaba scale.

Tuning playbook

Set ECN marking thresholds (K_min/K_max) at 80%/95% of buffer or vendor recommendation. Too low and you over-react to transient bursts; too high and PFC fires before ECN catches.
Tune R_AI (rate-increase step) to ~5 Mbps per RTT. Lower for sensitive workloads, higher for fast convergence.
Enable adaptive routing in switches. Static ECMP causes hash polarization.
Monitor PFC PAUSE counters. Non-zero per minute means tune or topology change.
Use jumbo frames (MTU 9000+). Reduces per-packet overhead.

When PFC catches fire

The failure mode: a single slow GPU causes its NIC to back-pressure its TOR, which back-pressures upstream switches, which back-pressure other TORs, until traffic on unrelated rails stalls. This is the canonical "PFC pause storm" and is what makes large RoCE clusters operationally hard. Defenses: per-flow congestion control (DCQCN/HPCC), buffer-aware adaptive routing, and conservative PFC enablement. Many production deployments now run "PFC-light" or "PFC-disabled with strong end-to-end CC" patterns.

LPO vs CPO economics and the 800G/1.6T transition

The optical layer is where a meaningful fraction of cluster power, capex, and failure rate lives. At 800G and 1.6T, the optical-module economics shift.

Pluggable optics (today's default)

QSFP56-DD (400G), OSFP (400G/800G), QSFP-DD800 (800G). Pluggable modules contain the DSP, drivers, and lasers. Pros: hot-swap, vendor flexibility, established supply chain. Cons: power (10-25W per 400G module), cost ($800-$2000 per 400G module, $1500-$4000 per 800G), and reach limits with copper.

DAC (Direct Attach Copper)

Passive copper cables. No optics. Lowest power, lowest cost, but limited to ~3m and not viable for racks-to-spine. Used heavily for in-rack and adjacent-rack connections.

AOC (Active Optical Cable)

Cable with optical modules pre-attached. Eliminates one connector pair vs separate cable + modules. Used for medium-reach (5-30m) in-row.

LPO (Linear-drive Pluggable Optics)

Removes the DSP from the module; relies on the host SerDes. Halves the module power and cuts ~40% of cost. Trade-off: tighter signal-integrity requirements, less reach margin. Production adoption began 2024-2025 for 800G; widely deployed for 1.6T 2026+.

CPO (Co-Packaged Optics)

Optics integrated into the switch ASIC package, eliminating per-port pluggables entirely. Massive power savings (potentially 50%+), lower latency, but trades hot-swap and vendor flexibility. Broadcom CPO roadmap targets production volume in 2026-2027; NVIDIA Spectrum-X CPO variants similar. Adoption gated by serviceability concerns.

Economics at scale

For a 32k-GPU cluster with 800G fabric: roughly 32k × 8 ports = 256k optical modules. At $2k/module (pluggable, 800G), that's $512M in optics — comparable to the GPU cost in some cases. LPO drops this by 40%; CPO by 60-70% with the trade-off that you replace whole switches rather than modules.

Optical layer comparison

Type	Power per 800G	Cost per port	Reach	Production 2026
Pluggable (QSFP-DD800)	16-25W	$1500-3000	100m-2km depending on optics	Default
LPO 800G	8-13W	$900-1800	Limited reach margin	Growing adoption
AOC 800G	12-18W	$800-1500	5-30m	In-row
DAC	< 1W	$200-500	< 3m	In-rack
CPO	est. 8-10W	est. $700-1200	Tied to switch lifetime	Limited, ramping

Why this matters for cluster planning

For a frontier-scale cluster, optical-layer choices dominate the operational cost line. LPO is the right default for 2026 800G+; CPO is the right default for 2027+ 1.6T deployments where serviceability concerns can be managed. DAC is best for in-rack; reserve pluggable optics for the parts of the topology where reach requires them.

Dual-plane fabric designs (Meta, Microsoft, OpenAI patterns)

At frontier scale, building a single non-blocking fabric is impractical. Production deployments split traffic across multiple planes to improve aggregate bandwidth, fault tolerance, and operational separation.

Meta SIGCOMM 2024 design

Meta's 24k-GPU H100 cluster paper (SIGCOMM 2024) describes a multi-plane fat-tree with 8 planes (one per rail), each rail terminating at one of 8 NICs per server. Each plane is a separate fat-tree with independent routing. Failure of one plane reduces fabric capacity by 12.5% but does not stop training. The design choice: rail-optimized topology with planes as the fault-tolerance unit.

Microsoft Azure ND H100/H200

Microsoft's ND H100 v5 and ND H200 v5 instances use a similar rail-optimized pattern with NVIDIA Spectrum-X. The Azure fabric has been documented as supporting up to ~100k GPUs with multi-plane design.

xAI Colossus

xAI's Memphis-based 100k-GPU Colossus cluster (publicly reported 2024) uses NVIDIA Quantum-2 InfiniBand with a multi-tier fat-tree. Few public details on the multi-plane structure; the scale of the deployment is the headline.

OpenAI / Stargate-class designs

Public details are limited, but Microsoft-OpenAI joint infrastructure references multi-data-center designs with cross-DC fabric. Stargate plans (Texas, 2025+) imply multi-GW power with novel networking. Speculation outpaces public confirmation.

Multi-plane benefits

Fault tolerance. Plane failure does not stop training.
Operational separation. One plane can be drained for maintenance while others run.
Parallel routing. Each plane has independent ECMP / adaptive routing, reducing hash polarization.
Scalability. Aggregate fabric capacity scales linearly with planes.

Multi-plane costs

More switches. N planes = N× switch count.
More optics. Linear scaling.
Routing complexity. Coordinating across planes is non-trivial.
NCCL topology hints required. Without correct topology info, NCCL may pick suboptimal algorithms.

Reference dual-plane example

A 16k-GPU H100 cluster: 16k × 8 rails = 128k NIC ports. Plane 1 connects rail 0 of every GPU; plane 2 connects rail 1; etc. Each plane is a 16k-port fat-tree, typically two-tier (TOR + spine) with 1.6:1 or 2:1 oversubscription. Eight independent planes.

Reference designs at 1k, 8k, 32k, 100k GPUs

Scale forces specific design choices. The following are operator-friendly defaults that can be tuned.

1k GPUs

Single-tier fat-tree fits. 128 servers × 8 GPUs = 1024 GPUs. With 64-port 400G InfiniBand switches (Quantum-2), one spine layer with 64 spine switches and 128 TOR-equivalent endpoints fits. Non-blocking, simple to operate. Cost dominated by GPU not fabric.

8k GPUs

Two-tier fat-tree. 1024 servers × 8 GPUs. Two-tier with rail-optimized design: 8 rails × 1024-port plane per rail. Some oversubscription (typically 2:1) is acceptable. SHARP enabled across spine. Tail latency starts to matter; tune NCCL aggressively.

32k GPUs

Three-tier fat-tree or dragonfly. 4096 servers × 8 GPUs. Multi-plane required (8 planes). Per-plane scale tests the limits of single-vendor fabrics. xAI and Meta operate at this scale and have publicly described their designs. Operational discipline (PFC tuning, telemetry, automated remediation) becomes critical.

100k GPUs

The frontier of 2026. Multi-DC required for power and cooling. Cross-DC fabric over WAN (1Tbps+ links). Hybrid fabric: InfiniBand within DC, dedicated DCI (Data Center Interconnect) between. xAI Colossus reportedly approaches 100k in a single Memphis facility; Stargate-class designs project beyond. Operational and physical-plant complexity dominate.

Per-scale cost breakdown

Scale	GPUs	Servers	Fabric tiers	Planes	Optical modules	Fabric cost share
1k	1024	128	1	1-2	~2k	5-10%
8k	8192	1024	2	8	~16k	10-15%
32k	32768	4096	3	8	~80k	15-25%
100k	100k+	12500+	3+ + WAN	8+	~250k+	20-30%

The fabric share grows superlinearly with scale; at 100k GPUs the fabric cost approaches 30% of the platform.

Failure-mode taxonomy and recovery patterns

Fabric failures are routine at scale. The question is how the cluster degrades.

Failure 1: Single link drop

A cable, transceiver, or NIC port fails. Detection: link-state-down event, NCCL retry. Recovery: rerun the affected collective on remaining paths (multi-path / adaptive routing helps), spare swap, or scheduled rebuild. Impact: minor if topology has redundancy.

Failure 2: Switch reboot

A switch (TOR or spine) reboots. Detection: all attached links drop. Recovery: NCCL retries via alternate paths if multi-plane; otherwise jobs stall. Impact: substantial if no redundancy.

Failure 3: PFC pause storm

Buffers fill, PFC pauses cascade. Detection: PAUSE counters spike, latency tail explodes. Recovery: drain traffic, identify offending flow, restart endpoint or kill flow. Impact: cluster-wide stall until storm resolves.

Failure 4: Congestion collapse

DCQCN parameters too aggressive, persistent low utilization despite congestion. Detection: low throughput, high queue depth, ECN marks at maximum. Recovery: retune DCQCN, may require restart of jobs. Impact: prolonged underperformance.

Failure 5: NIC firmware bug

NIC silently drops packets or corrupts. Detection: NCCL hash mismatches, training divergence. Recovery: rolling firmware update, NIC replacement. Impact: data corruption, training restart from checkpoint.

Failure 6: ECMP hash polarization

Multiple flows hash to the same path, creating hot spot. Detection: per-port utilization imbalance. Recovery: adaptive routing, hash-key rotation, or explicit path assignment. Impact: tail latency degradation.

Failure 7: Optical layer (LPO marginal signal)

LPO marginal signal-integrity at temperature extremes. Detection: BER spikes correlated with environmental sensors. Recovery: adjust thresholds, replace marginal modules. Impact: intermittent loss, slow tail latency.

Failure 8: Cross-rack link saturation

In multi-rack training, inter-rack bandwidth becomes the bottleneck. Detection: per-rack collective time scales differently. Recovery: topology-aware placement, oversubscription reduction. Impact: 1.5-3× collective time on affected paths.

Failure 9: Slow GPU dragging fabric

A GPU runs slowly (thermal, ECC errors). Its NIC ends up back-pressured. Fabric appears slow because the slowest rank is slow. Detection: per-rank timing skew. Recovery: drain the slow GPU, replace, restart. Impact: cluster-wide stall.

Failure 10: Routing-protocol convergence

BGP/OSPF reconvergence after link flap takes seconds. Detection: brief packet loss, NCCL retries. Recovery: BFD tuning, smaller failure domains. Impact: small but cumulative.

Recovery patterns

Hot-standby: spare GPUs and links pre-allocated for failover.
Cold-standby: failures drain to checkpoint, restart on healthy fabric.
Multi-plane: plane failure does not stop training.
Telemetry-driven: automated detection + remediation.
Periodic rebuilds: scheduled drain + repair windows.

Cross-DC training over WAN

When a single data center cannot supply power for a training run, training spans multiple facilities. The fabric extends over wide-area links.

Power constraints driving cross-DC

A 100k-GPU Blackwell cluster consumes 130-200 MW. Few data centers have this much power available; new facilities take 18-36 months to build. Cross-DC training is the bridge.

WAN fabric characteristics

Bandwidth: 1-10 Tbps between paired DCs is now economically feasible with dedicated dark fiber.
Latency: 1-10 ms RTT for paired DCs in the same region; cross-region adds 30-100ms.
Reliability: dedicated dark fiber has high availability but failures are recovery-from-snapshot expensive.

Workload partitioning for cross-DC

Pipeline-parallel training maps naturally across DCs because pipeline stages are serial. Data-parallel across DCs is brutal because all-reduce becomes WAN-bound. Mixed designs: DP within DC, PP across DC, with optimizer-state sharding (ZeRO-3) tuned to minimize cross-DC traffic. See distributed LLM training.

Disaggregated-inference cross-DC

Cross-DC disaggregated inference (prefill in one DC, decode in another) is being explored but adds latency to time-to-first-token. See disaggregated inference for the underlying mechanics.

Stargate / Hyperion-class designs

OpenAI Stargate, Microsoft Hyperion, Amazon Project Rainier (publicly reported) all imply multi-DC fabrics. Specifics are not public; the order of magnitude is "multiple GW power across paired DCs."

Cross-DC failure modes

WAN link failure is rare but catastrophic when it happens. Recovery: restart from checkpoint with mid-flight gradient state lost. Cluster-wide stall until WAN recovers.

Storage networking on the same fabric (NVMe-oF)

The same InfiniBand or RoCE fabric typically carries checkpoint traffic. Sharing the fabric has consequences.

NVMe-over-Fabrics (NVMe-oF)

NVMe-oF runs NVMe semantics over RDMA. Storage targets (DGX SuperPOD storage, all-flash arrays from VAST, Pure Storage, DDN, WekaIO) export NVMe namespaces over IB or RoCE. Bandwidth: 400Gbps per port, latency: 10-100μs to remote storage.

Checkpoint traffic on training fabric

Checkpoints for a 70B model are roughly 280 GB (FP32) or 140 GB (FP16). Saving every N steps at 100 GB/s (saturating a few NICs) takes ~3 seconds, which is acceptable. The wrinkle: checkpoint traffic shares the fabric with collectives. If checkpoint and all-reduce coincide, both slow down.

Mitigations

Dedicated checkpoint plane. A subset of the fabric reserved for storage traffic. Adds cost.
Checkpoint scheduling. Coordinate with training step boundaries so storage traffic happens during compute, not collective.
Asynchronous checkpoints. GPU writes to CPU memory; CPU writes to remote storage in background.
Tiered storage. Local NVMe for fast checkpoints; periodic uploads to remote.

See checkpoint storage and recovery for the broader pattern.

Per-cloud network reality (AWS EFA, Azure, GCP, Lambda, CoreWeave)

Each cloud has made specific choices that affect network behavior. Practical detail.

AWS

EFA + SRD on P5 / P5e (H100) and P5en / P6 (B200) instances. SRD is packet-sprayed unreliable transport; reliability and ordering reassembled at the NIC. NCCL plugin is mature. UltraClusters of 32k+ H100 documented. No public InfiniBand offering. Cross-AZ latency 1-2ms; intra-AZ < 100μs typical.

Azure

ND H100 v5 and ND H200 v5 use NVIDIA Quantum-2 InfiniBand for compute traffic, Spectrum-X Ethernet for variants. Hundreds-of-thousands of GPUs across regions. Confidential GPU VMs available.

GCP

A3 / A3 Mega / A4 instances use Falcon-based RDMA over Ethernet. Highly tuned NCCL. TPU pods use Inter-Chip Interconnect (ICI) which is a custom torus fabric, not Ethernet-like.

Lambda Labs

H100 / H200 / B200 clusters using NVIDIA Quantum-2/3 InfiniBand. Targets AI-specific workloads with simpler procurement than hyperscalers.

CoreWeave

H100 / H200 / B200 with NVIDIA Quantum-2/3 InfiniBand. Multi-region deployments. Used by Microsoft for OpenAI inference burst capacity.

Oracle OCI

H100 / H200 / B200 with NVIDIA Quantum InfiniBand. Some confidential compute by default.

Per-cloud fabric comparison

Cloud	GPU fabric	Per-port speed	Per-cluster scale	RDMA model
AWS UltraCluster	EFA + SRD (Ethernet)	400G	32k+	SRD (packet-sprayed)
Azure ND H200	Quantum-2 IB	400G	100k+	RC RDMA
GCP A3+	Falcon (Ethernet)	varies	tens of k	Falcon hardware-reliable
Lambda	Quantum IB	400G/800G	tens of k	RC RDMA
CoreWeave	Quantum IB	400G/800G	tens of k	RC RDMA
Oracle	Quantum IB	400G/800G	tens of k	RC RDMA

Migration friction

Moving training jobs between clouds requires retuning NCCL, sometimes recompiling against vendor-specific transport plugins. Multi-cloud training is uncommon for this reason; multi-cloud inference is more tractable because per-call traffic patterns are simpler.

2026 frontier-cluster networking case studies

A few public-record deployments illustrate the state of the art.

xAI Colossus (Memphis, 100k H100)

Publicly reported as the largest single-facility H100 cluster as of late 2024. NVIDIA Quantum-2 InfiniBand. Specific topology details are not public; the scale and the speed of deployment are the headline.

Meta Llama-3.1 training cluster

Meta's SIGCOMM 2024 paper on their 24k-H100 cluster documents an Ethernet (RoCE)-based design with rail-optimized fat-tree, multi-plane structure, and tuned DCQCN. Meta has publicly discussed Llama-3.1 training on similar fabric. Notable: a major hyperscaler running RoCE at frontier scale by 2024.

Microsoft Azure / OpenAI training capacity

Multiple Azure ND H100 v5 and ND H200 v5 deployments serve OpenAI training. Public scale: hundreds of thousands of GPUs across regions. NVIDIA Quantum-2 IB. Cross-DC capacity emerging.

AWS UltraClusters

UltraCluster announcements through 2024-2025 describe 32k+ H100 instances on EFA-SRD Ethernet. Used by Anthropic, Adept, Stability, and others. AWS doubles down on Ethernet rather than InfiniBand.

Google A3 / A4

GCP A3 Mega and A4 clusters serve internal training (Gemini) and external customers. Falcon-based fabric. TPU pods (separate from A3/A4) use ICI custom interconnect for the TPU v5p, v6 generations.

Stargate (forthcoming)

OpenAI / Microsoft Stargate is the project that consolidates frontier-training capacity for the late-decade. Public details suggest multi-GW, multi-DC, with novel fabric designs. Specifics not public.

What these case studies show

InfiniBand still dominates at very large scale, but Ethernet (RoCE, Falcon, EFA-SRD) is operational at 32k+ scale. UEC will likely accelerate Ethernet adoption. CPO and LPO economics will reshape optical costs. Cross-DC is the next frontier.

Additional FAQ

Q: Does Spectrum-X match InfiniBand for frontier training?

Spectrum-X closes most of the per-collective gap when paired with ConnectX-7/8 and BlueField-3, but in production benchmarks InfiniBand with SHARP still leads on small-message all-reduce by 15-30%. For large-message all-reduce the gap narrows to under 10%. Operationally Spectrum-X requires substantially more tuning effort; Quantum-2/3 is more turnkey.

Q: Is 1.6T Ethernet shipping in 2026?

Yes, in early production. Broadcom Tomahawk-6 and NVIDIA Spectrum / Quantum next-gen are publicly announced. Early deployments are 2026; broader availability 2027. 1.6T per port doubles 800G, which is the current default for new frontier deployments.

Q: How does adaptive routing compare to ECMP?

ECMP uses a fixed flow-based hash; all packets in a flow take the same path. Hash polarization (many flows mapping to one path) is the classic failure mode. Adaptive routing decides per-packet (or per-flowlet) based on real-time switch state, dodging hot spots. InfiniBand has adaptive routing built in; Ethernet adds it through Spectrum-X or vendor features. Adaptive routing is essential at frontier scale; ECMP-only is fine for smaller clusters.

Q: What's the operational benefit of NVL72 over multiple 8-GPU servers?

NVL72 makes 72 GPUs look like one giant GPU on the NVLink fabric. All-reduce across 72 GPUs happens at ~1.8 TB/s per GPU on NVLink, vs ~50 GB/s per GPU on InfiniBand. For MoE serving with large EP groups, this is transformative. See NVLink and rack-scale topology for the underlying fabric.

Q: Can I run an 800G fabric with passive DAC?

Only within a rack (< 3m). For racks-to-spine you need active optics: AOC, pluggable, or LPO. The economics of 800G heavily favor LPO for new builds.

Q: How does congestion control interact with the all-reduce algorithm choice?

Ring all-reduce has steady, bursty long-running flows; tree all-reduce has shorter, bursty flows; SHARP has compressed in-network reduction. Congestion control reacts differently to each. DCQCN and HPCC are tuned for steady flows; they over-react to short bursts. Tuning per workload helps.

Q: What's the role of jumbo frames?

Jumbo frames (MTU 9000+) reduce per-packet header overhead. For RDMA on RoCE this is helpful at 400G+. For InfiniBand it's the default. Mismatched MTU between endpoints causes silent fragmentation or PMTU discovery delays — keep it consistent across the cluster.

Q: How do I monitor fabric health in 2026?

Telemetry stacks: NVIDIA UFM for InfiniBand, NetQ + What-Just-Happened for Spectrum-X, Cisco / Arista vendor stacks for Ethernet. Key signals: PFC PAUSE counters, ECN marks, CRC errors, link flap rate, NCCL collective latency histograms. Aggregate dashboards correlate fabric metrics to job throughput.

Q: Does AWS EFA support NCCL out of the box?

Yes, via the AWS-OFI-NCCL plugin (open source). Production-tuned for P5 / P5e instances. Mature; minimal manual tuning required vs raw RoCE.

Q: What's the best in-rack fabric for B200?

NVL72 with NVLink + NVSwitch fully populated. Within the rack, NVLink is the fabric. Cross-rack uses InfiniBand or RoCE. The NVL72 architecture changes the unit of scaling: one NVL72 is "one giant accelerator."

Q: How does cross-DC training latency affect step time?

Cross-DC RTT is 1-10ms for paired DCs. Synchronous DP across DCs is impractical; cross-DC training uses pipeline-parallel partitioning so collective traffic is intra-DC. Some optimizer steps can absorb cross-DC latency via async or low-frequency communication.

Q: Is there a standard for cross-DC AI fabric?

Not yet. Each hyperscaler builds bespoke DCI for their AI fabrics. Standardization is an open area; UEC may eventually extend.

Q: How do I size oversubscription for an AI cluster?

For pure training: 1:1 (non-blocking) is the safe default but most expensive. 2:1 oversubscription is acceptable for well-tuned all-reduce. Higher than 2:1 hurts tail latency. For inference: 4:1 or higher is fine because traffic patterns are simpler.

Q: What's the operational difference between SRD and RC RDMA?

RC (Reliable Connection) RDMA maintains per-connection state and in-order delivery. SRD (Scalable Reliable Datagram, AWS) drops the connection state, sprays packets across paths, and reassembles at the NIC. SRD scales better (no per-flow state on switches) but breaks some assumptions of legacy code. NCCL is adapted for SRD via the AWS plugin.

Q: When is RoCE clearly better than InfiniBand?

Operational integration with existing Ethernet infrastructure, vendor flexibility (avoid NVIDIA lock-in), cost per port at scale, and unified networking with storage/management. For frontier training, InfiniBand still leads on tail latency; RoCE catches up with discipline and UEC v1.0.

Table of contents

Key takeaways

Mental model: AI cluster networking in one minute

The networking landscape in 2026

Quick-reference: protocols at a glance

The networking layers

Layer 1: intra-node (NVLink)

Layer 2: inter-node (InfiniBand or RoCE)

Layer 3: management plane (Ethernet)

NVLink and NVSwitch

NVLink versions

NVSwitch

Why NVLink matters

InfiniBand

Bandwidth tiers

Per-rail vs per-rack bandwidth

Latency

Operational notes

RoCE (RDMA over Ethernet)

Why RoCE

Why not RoCE

When RoCE works well

When RoCE struggles

Cloud RoCE

Topology: rail-optimized vs fat-tree

Rail-optimized (NVIDIA's recommendation)

Fat-tree

Modern frontier clusters

Rail-optimized in practice: the 1024-GPU reference design

Why dual-plane topologies became popular in 2024-2025

Bisection bandwidth: what it is, why it matters

Per-rail bisection

When bisection matters

Per-collective bandwidth requirements

Tail latency at scale

Diagnosing fabric health

nccl-tests

IB diagnostics

Continuous monitoring

GB200 NVL72: rack-scale NVLink

What changes

When this matters

When it doesn't matter

Cost

Cloud networking variants

AWS

GCP

Azure

Specialized GPU clouds (CoreWeave, Lambda, Crusoe)

Picking a cloud

A short history of AI training networking

NCCL collective performance characteristics

All-reduce (DP gradients, TP activations)

All-gather (FSDP parameters)

Reduce-scatter (FSDP gradients)

All-to-all (MoE routing)

Broadcast

Tree vs Ring algorithms

Topology choice in detail

Rail-optimized

Fat-tree

Dragonfly+

Cluster-of-clusters

Per-node bandwidth math

Switch architectures

NVIDIA Quantum-2 (NDR)

Quantum-3 (XDR)

Cisco/Arista/Mellanox Ethernet for RoCE

NCCL on RoCE: gotchas

GID confusion

MTU mismatch

PFC not propagated

Buffer credits

Fabric monitoring in production

Per-port metrics

Per-host metrics

Periodic benchmarks

Optical interconnect future

Why optics

Today