NVIDIA Datacenter GPUs for AI: The Complete Guide

Q: Should I buy H100 or H200 for inference?

H200 if your workload is long-context-heavy (32k+) or memory-bandwidth-bound. H100 if it's compute-bound or moderate-context.

Q: When does Blackwell make sense?

For new frontier training runs and for the highest-throughput inference workloads. Lower priority for cost-optimized inference and research.

Q: What about A100? Still useful in 2026?

Yes, in cost-optimized inference deployments where H100/H200 don't justify the price. A100s are still the cheapest GPU capacity, especially on spot.

Q: GB200 vs 9× DGX H100 — which is better for frontier training?

GB200 wins for very large models requiring TP > 8. Otherwise comparable; 9× DGX H100 is more flexible (can split into 8-GPU islands per workload).

The decision between H100, H200, B200, and GB200 isn't about "which is fastest" — that's settled (Blackwell wins). It's about cost, availability, workload fit, and operational maturity. Most teams in 2026 still run H100s for production and reserve B200s for new training runs. This guide explains why. If you are starting further back, what a GPU is and why AI needs them covers the fundamentals this hardware guide builds on.

Key takeaways
Mental model: GPU architecture in one minute
Family tree: A100 → Hopper → Blackwell → Rubin
Hopper (H100, H200)
Blackwell (B100, B200, GB200)
Spec sheet comparison
HBM: bandwidth and capacity
Tensor cores: FP16, FP8, FP4
NVLink and node topology
Power and thermals
The Rubin family (2026–2027)
Pricing and availability
Workload fit: training vs inference
Picking the right SKU
Capacity planning examples
Arithmetic intensity and roofline
Per-workload performance ceilings
Total cost of ownership math
Hardware-software gotchas by generation
H200 vs B200 for inference
When not to upgrade
Power and cooling: from air to liquid
NCCL and GPU-to-GPU communication
Hopper-to-Blackwell migration tips
Rubin family preview: R100, GR200, and rack-scale plans
Secondhand pricing trajectory
CUDA compatibility across generations
Per-SKU deep dive: every datacenter GPU NVIDIA ships
NVLink, NVSwitch, and PCIe generation history
Multi-Instance GPU (MIG) in detail
Tensor Memory Accelerator and TCGen5
Multi-vendor comparison: AMD, TPU, Trainium, others
GB200 NVL72: rack-scale engineering
Export controls: H800, H20, B30 China-market variants
Cloud availability matrix: AWS, Azure, GCP, specialist
Workload-to-SKU map: what to use for what
The bottom line
FAQ
Extended FAQ
Glossary
References

Key takeaways

Hopper (2022–2025) is still the workhorse:

H100 80GB: the standard. 3.0 TB/s HBM, 1979 TFLOPs FP16, 989 TFLOPs FP32, 3958 TFLOPs FP8.
H200 141GB: H100 with bigger, faster HBM. Same compute, 4.8 TB/s, ~1.7× HBM capacity. Inference winner.

Blackwell (2024–) is the new frontier:

B100 192GB: cooler-running variant of B200. Common in retrofit deployments.
B200 192GB: the headline. ~5× FP4 throughput vs H100 FP8, 8 TB/s HBM, NVLink-5.
GB200 NVL72: 72-GPU rack with NVSwitch fabric. The "one big GPU" pitch.

Rubin (announced 2026): GB300 / R100 succeeds Blackwell in late 2026 / 2027. ~2× perf-per-watt vs Blackwell, native FP4-everywhere.

Picking rule of thumb:

Training new frontier model: B200 or GB200.
Inference at scale, long context: H200 or B200.
Inference at scale, moderate context: H100 (cheapest available capacity).
Research/prototyping: H100 (deep spot market, mature ecosystem).

Mental model: GPU architecture in one minute

The named problem is the arithmetic-intensity wall: a modern GPU can do 10-50× more flops per second than its HBM can feed it bytes. An H100 delivers ~2000 TFLOPs of FP16 but only 3.0 TB/s of memory bandwidth — roughly 660 flops per byte loaded. If your kernel does fewer flops per byte than that ratio, you are stalled on memory, the tensor cores idle, and buying more flops is wasted money. Every architecture generation since Volta is essentially a war on this ratio.

The core idea is the same one CPUs solved with caches, escalated. NVIDIA attacks the wall on three fronts simultaneously: bigger and faster HBM (80 GB at 3.0 TB/s on H100 → 141 GB at 4.8 TB/s on H200 → 192 GB at 8.0 TB/s on B200), lower-precision tensor cores (FP16 → FP8 on Hopper → FP4 on Blackwell, doubling effective flops per byte at each step), and fatter NVLink so that a "GPU" is really an 8- or 72-way coherent fabric (NVLink-4 at 900 GB/s on H100, NVLink-5 at 1.8 TB/s on B200, NVL72 at rack scale). Think of a roofline chart: HBM raises the slanted memory roof, FP8/FP4 raises the flat compute roof, NVLink raises the cross-GPU roof.

Generation	HBM cap / BW	Dense FP16 TFLOPs	Lowest precision / TFLOPs	NVLink BW (per GPU)	Where the wall moved
A100	80 GB / 2.0 TB/s	312	TF32 / BF16	600 GB/s	Compute-bound
H100	80 GB / 3.0 TB/s	989	FP8 / 3958	900 GB/s	FP8 raised compute roof
H200	141 GB / 4.8 TB/s	989	FP8 / 3958	900 GB/s	HBM raised memory roof
B200	192 GB / 8.0 TB/s	~2250	FP4 / ~9000	1.8 TB/s	FP4 + NVL72 raise both

Conceptually a kernel decides which roof it hits:

flops_per_byte = kernel_flops / kernel_bytes
if flops_per_byte < gpu.flops / gpu.hbm_bw:   # ~660 on H100
    bound = "memory"   # H200 helps, FP8 doesn't
else:
    bound = "compute"  # FP8/FP4 helps, H200 doesn't

One number to remember: decode-phase LLM inference on H100 runs at ~10-30 flops per byte — deeply memory-bound — which is exactly why H200's extra HBM bandwidth gives ~1.6-1.9× decode throughput at the same compute.

The rest of this guide is everything that extends or depends on that idea — generation-by-generation specs, when each SKU pays back, and the TCO math behind picking one.

Family tree: A100 → Hopper → Blackwell → Rubin

NVIDIA's datacenter GPU cadence:

2020 — A100 (Ampere): 80 GB HBM2e, 312 TFLOPs TF32, 1248 TFLOPs FP16. The training platform for GPT-3, early Llama. Now end-of-life for new deployments but still common in inference fleets.
2022 — H100 (Hopper): 80 GB HBM3, 1979 TFLOPs FP16, 3958 TFLOPs FP8. Doubled FP16 throughput, introduced FP8 (4× peak FP16). Trained Llama-3, Claude 3, GPT-4 era.
2023 — H200 (Hopper): same compute as H100, 141 GB HBM3e, 4.8 TB/s. Inference-focused refresh.
2024 — B100 / B200 (Blackwell): 192 GB HBM3e, 8 TB/s, 18 PFLOPs FP4. Major architectural shift.
2024 — GB200 NVL72: rack-scale Blackwell with 72 GPUs in one NVLink fabric. Treats the rack as one unit.
2026 — R100 / GB300 (Rubin): announced. ~2× perf-per-watt vs Blackwell, ~3× HBM bandwidth.

Each generation roughly doubles peak compute and increases HBM. The pattern is consistent: NVIDIA ships the chip, the field adopts it 12–18 months later in production.

Hopper (H100, H200)

Why Hopper mattered

The transition from Ampere (A100) to Hopper (H100) wasn't incremental — it was the first GPU generation explicitly designed for transformer workloads. Three architectural innovations:

Transformer Engine: hardware-software stack that automatically manages FP8 quantization for matrix multiplies. Enables 2× throughput vs BF16 with minimal quality cost. The FP8 formats themselves (E4M3 and E5M2) were standardized in arXiv:2209.05433, building on the earlier mixed-precision training playbook from arXiv:1710.03740. See our mixed-precision training guide for the numerics.

Thread Block Cluster: a new programming primitive that groups multiple thread blocks into clusters with shared memory access. Useful for transformer-style operations where multiple blocks process related data.

HBM3: doubled bandwidth vs HBM2e on A100. HBM3 became HBM3e on H200 and B-series, doubling again.

The combination meant Hopper roughly tripled real-world transformer throughput vs Ampere, while also enabling new memory-intensive workloads (long-context inference, very large batched training).

H100 — the workhorse

The dominant GPU for AI workloads from 2023 through ~2025. Three SKUs:

H100 SXM 80GB: NVLink-connected, used in DGX systems. The training-grade variant.
H100 PCIe 80GB: PCIe-attached, for bring-your-own-server. Slightly lower power, no NVLink between H100s.
H100 NVL 94GB: dual-board variant for inference, 94 GB HBM total.

Key specs:

Compute: 1979 TFLOPs FP16, 3958 TFLOPs FP8 (peak).
HBM: 80 GB HBM3, 3.0 TB/s.
TDP: 700W (SXM) / 350W (PCIe).
NVLink-4: 900 GB/s/GPU bidirectional.

What changed vs A100: doubled tensor core throughput, native FP8 support (with Transformer Engine), faster HBM (3 TB/s vs 1.5 TB/s).

H200 — Hopper for inference

Same compute, bigger HBM:

141 GB HBM3e (vs H100's 80 GB).
4.8 TB/s (vs 3.0 TB/s).
TDP: 700W.

The story: H200 is a Hopper die with a memory upgrade. Compute throughput is identical to H100. The 1.7× HBM capacity makes it dramatically better for inference workloads where memory bandwidth is the bottleneck (decode) or where capacity matters (long context).

H100 vs H200 for typical inference:

Llama-3 70B at 32k context: H100 fits ~3 concurrent, H200 fits ~5.
Llama-3 70B decode tokens/sec: H100 ~30/sec, H200 ~45/sec (proportional to bandwidth).

For training-only workloads, H200 doesn't beat H100 enough to justify the price premium.

Hopper-specific software innovations

FlashAttention-2 and FlashAttention-3: kernel-level optimizations for attention that work especially well on Hopper. FA-2 (arXiv:2307.08691) reworked the parallelism strategy across warps; FA-3 (arXiv:2407.08608) specifically uses Hopper's TMA (Tensor Memory Accelerator) and warp-specialized async pipelines for ~75% of peak FP16 throughput. We unpack the kernel implications in our long-context attention guide and Triton kernel primer.

Transformer Engine library: NVIDIA's library for FP8 training and inference. Automatically handles per-tensor scaling, calibration, and the BF16 fallback for numerically-sensitive layers.

CUDA Graphs: capture a sequence of CUDA operations and replay them with minimal CPU overhead. Particularly useful for inference where the same forward pass repeats.

These aren't strictly Hopper features — they work on any post-Ampere GPU — but they're optimized specifically for Hopper's architecture.

H100 NVL — the inference-optimized variant

H100 NVL is a dual-board variant: two H100 dies on one PCIe card with 94 GB combined HBM. Used in some inference deployments where:

Maximum HBM per dollar matters.
NVLink between the two halves is sufficient (no need for full-fat NVSwitch).

H100 NVL has gone niche by 2026; H200 SXM offers similar capacity with better integration.

Blackwell (B100, B200, GB200)

B200 — the headline

NVIDIA's 2024 frontier accelerator. Major architectural changes:

Dual-die package: two reticle-limit dies fused with a 10 TB/s interconnect. Software sees one GPU.
HBM3e: 192 GB total (96 GB per die), 8 TB/s.
Native FP4 tensor cores: 18 PFLOPs FP4 (vs 4 PFLOPs FP8 on H100, ~5× peak per chip).
NVLink-5: 1.8 TB/s/GPU bidirectional.
TDP: 1000W.

The big shift is FP4. Hopper's FP8 tensor cores were introduced in 2022; FP4 doubles them again. For training in FP4 (still emerging in 2026), Blackwell offers ~2× the throughput of equivalent Hopper deployments.

B100 — the lower-power sibling

B100 has the same architecture as B200 but lower clocks and lower TDP (~700W). Used in retrofits where the existing infrastructure can't handle B200's 1000W. Compute is ~75% of B200.

GB200 NVL72 — rack-scale

The pitch: 72 Blackwell GPUs in one rack, all connected via NVLink-5 through NVSwitch. The entire rack acts as one giant GPU for software purposes — TP can span all 72. We cover this fabric in depth in NVLink and rack-scale topology and the parallelism trade-offs in distributed LLM training.

Key numbers:

72 B200 GPUs per rack.
1296 PFLOPs FP4 peak per rack.
13.5 TB total HBM3e.
130 TB/s aggregate HBM bandwidth.

This is for frontier training where TP=72 within one fabric is required. Most teams don't need this; standard 8-GPU NVLink islands are enough.

Why Blackwell mattered

Hopper made transformers fast. Blackwell made them dramatically faster, primarily through:

Native FP4 tensor cores: 2× FP8 throughput. For inference workloads that can tolerate FP4 quality cost, Blackwell offers significant capacity-per-dollar improvement.

Dual-die design: instead of pushing reticle limits with one bigger die, NVIDIA used two dies stitched with a 10 TB/s interconnect. Software sees one logical GPU. This let them double effective compute while staying within manufacturable die sizes.

HBM3e at 8 TB/s: another 1.7× bandwidth jump vs H200. Critical for keeping FP4 tensor cores fed.

NVLink-5 at 1.8 TB/s: 2× NVLink-4. Makes the GB200 NVL72 rack-scale architecture practical — you can run TP=72 across the rack at speeds that don't murder collective time.

The result: Blackwell B200 delivers ~2.5× the effective LLM training throughput of H100 (FP8 → FP4 with calibration), and ~3-4× the inference throughput (HBM bandwidth + FP4 + better KV cache management).

Blackwell HGX vs DGX

NVIDIA ships Blackwell in two reference platforms:

HGX B200: 8-GPU baseboard. OEMs (Dell, HPE, Supermicro) build full systems around it. Same compute as DGX but more flexibility in chassis design.

DGX B200: NVIDIA's first-party 8-GPU system. Reference design, NVIDIA support, premium pricing.

For on-prem deployments, HGX through an OEM is usually 20-30% cheaper. For "I want NVIDIA's blessing for my deployment," DGX is the answer.

B200 in production

By mid-2026, B200 deployments are widespread but tight on supply:

Frontier labs (Meta, Google, Anthropic, OpenAI) prioritize it for new training runs.
Major cloud providers (AWS, Azure, GCP) have B200 instances available with reservations.
Specialized GPU clouds (CoreWeave, Lambda) have B200 fleets for specific customer commitments.
Smaller deployments mostly still use H100 due to price/availability.

Expect B200 to be the standard "premium" SKU through 2027, with H100 remaining the cost-optimized choice and Rubin (R100/GB300) emerging.

Spec sheet comparison

Spec	A100 80GB	H100 SXM	H200 SXM	B100 SXM	B200 SXM
HBM capacity	80 GB	80 GB	141 GB	192 GB	192 GB
HBM bandwidth	1.5 TB/s	3.0 TB/s	4.8 TB/s	7.7 TB/s	8.0 TB/s
FP16 tensor	624 TFLOPs	1979 TFLOPs	1979 TFLOPs	3500 TFLOPs	4500 TFLOPs
FP8 tensor	n/a	3958 TFLOPs	3958 TFLOPs	7000 TFLOPs	9000 TFLOPs
FP4 tensor	n/a	n/a	n/a	14000 TFLOPs	18000 TFLOPs
NVLink	NVLink-3, 600 GB/s	NVLink-4, 900 GB/s	NVLink-4, 900 GB/s	NVLink-5, 1.8 TB/s	NVLink-5, 1.8 TB/s
TDP	400W	700W	700W	700W	1000W
Process	TSMC N7	TSMC 4N	TSMC 4N	TSMC 4NP	TSMC 4NP

(Note: SXM versus PCIe variants differ slightly; numbers above are SXM/datacenter-grade. PCIe is typically 80% of these.)

Compute throughput claim corrections

Marketing slides love big numbers. Real-world sustained throughput is ~50-70% of peak FLOPs because:

Tensor cores aren't fully utilized except in best-case matmuls.
Memory bandwidth limits decode and small-batch operations.
Power throttling on sustained workloads.

Plan with sustained numbers, not peak.

HBM: bandwidth and capacity

HBM is the single most consequential spec for AI workloads. Two dimensions:

Capacity

How much can you fit on one GPU? Determines:

Largest model you can serve without TP.
Maximum context × batch you can hold for one model.

Capacity progression:

A100: 80 GB.
H100: 80 GB.
H200: 141 GB.
B100/B200: 192 GB.
GB300/R100 (announced): 256 GB.

H200 was a big jump (+76% over H100). B200 added ~36% more on top.

Bandwidth

How fast can you read/write HBM? Determines:

Decode token rate (memory-bandwidth-bound).
Activation movement during forward/backward.
KV cache read efficiency.

Bandwidth progression:

A100: 1.5 TB/s.
H100: 3.0 TB/s.
H200: 4.8 TB/s.
B200: 8.0 TB/s.
R100 (announced): ~13 TB/s.

Each generation roughly 60-70% bandwidth bump. This compounds: B200's ~5× FP4 throughput is only useful if HBM can feed it, which Blackwell's 8 TB/s addresses.

Why bandwidth matters more for inference

Decode reads the entire model + KV cache per token. For Llama-3 70B on H100:

140 GB / 3 TB/s = 47ms per layer per token.
~30 tokens/sec achievable.

Same model on H200:

140 GB / 4.8 TB/s = 29ms.
~45 tokens/sec.

That's a 1.5× speedup just from HBM bandwidth. For training, HBM matters less (compute-bound); for inference, it's everything.

Tensor cores: FP16, FP8, FP4

Tensor cores are matrix-multiply units. They accelerate the dominant operation in transformers (matrix multiplies). Each generation adds new precision formats.

FP16 / BF16

The pre-2022 default. Both have 16-bit width but different exponent/mantissa splits:

FP16: 5 exp, 10 mantissa. Narrow dynamic range, fine precision.
BF16: 8 exp, 7 mantissa. Same dynamic range as FP32, coarser precision.

BF16 is the standard for modern training because its dynamic range matches FP32 (no overflow during gradient accumulation). FP16 is faster on some older hardware but needs gradient scaling.

FP8

Introduced on Hopper (H100). Two formats:

e4m3: 4 exp, 3 mantissa. For activations/KV cache.
e5m2: 5 exp, 2 mantissa. For gradients during training.

Throughput on H100: 3958 TFLOPs (vs 1979 TFLOPs FP16). 2× speedup.

Quality: ~0.1 point on MMLU vs BF16 with proper calibration.

FP4

New on Blackwell. e2m1 format. Throughput on B200: 18 PFLOPs.

Quality: still being characterized. Early data shows ~0.5–1 point quality cost on standard benchmarks vs FP8. Better with sophisticated calibration.

FP4 is most compelling for inference (where quality cost is more easily tolerated) and emerging for training (where loss curves need careful watching).

What about INT8 / INT4?

Integer quantization is a separate path:

INT8: similar throughput to FP8, simpler hardware support.
INT4: aggressive quantization for memory-bound deployments.

Tensor cores natively support these formats. Quality cost is similar to FP variants.

For most production deployments in 2026: FP8 weights + FP8 KV is the modern default on Hopper, FP4 + FP8 is emerging on Blackwell.

NVLink and node topology

NVLink is the inter-GPU interconnect within a node. Critical for tensor-parallel training and inference.

Versions

NVLink-3 (A100): 600 GB/s/GPU.
NVLink-4 (H100, H200): 900 GB/s/GPU.
NVLink-5 (B100, B200): 1.8 TB/s/GPU.

Topology within a node

8-GPU server (DGX H100):

All-to-all NVLink fabric via NVSwitch.
Any GPU can talk to any other at full NVLink speed.
TP=8 within one node is essentially "free" from a communication perspective.

GB200 NVL72:

72 GPUs across 18 boards in one rack.
NVSwitch fabric connects all 72 GPUs at NVLink-5 speeds.
TP=72 within one rack. Frontier training only.

Why NVLink matters

For TP, every transformer layer requires an all-reduce. On NVLink (900 GB/s), this is microseconds. Across InfiniBand (50 GB/s), it's milliseconds. Difference is 20×.

This is why TP rarely scales past 8 in practice: NVLink stops at the node boundary. Going across nodes via InfiniBand makes TP communication-dominated.

GB200's rack-scale NVLink fabric extends this to 72 GPUs. For frontier training, this matters. For most workloads, 8-GPU islands are sufficient.

Power and thermals

H100: 700W TDP. Standard datacenter cooling handles it.

B200: 1000W TDP. Pushes the limits of air cooling. Many B200 deployments use liquid cooling, especially in dense rack configurations.

GB200 NVL72: ~120 kW per rack. Liquid cooling required. Datacenter facilities need to be designed for this density — many existing facilities can't house GB200 racks without retrofit.

The takeaway: B200 deployment requires datacenter readiness. Cloud providers (CoreWeave, Lambda, AWS) handle this; on-prem deployments need infrastructure investment.

The Rubin family (2026–2027)

Announced at GTC 2024, expected late 2026 / early 2027:

R100: successor to B200. ~2× perf-per-watt, ~3× HBM bandwidth (estimated).
GB300 NVL72: rack-scale Rubin successor to GB200.
NVLink-6: ~3.6 TB/s/GPU expected.
HBM4: ~1 TB/s/stack (vs HBM3e's ~640 GB/s).

Specs aren't fully public yet. The architectural direction:

More aggressive FP4 throughput.
Native FP6 for training (compromise between FP4 speed and FP8 quality).
Co-packaged optics for inter-rack interconnect (replacing some InfiniBand).

For planning purposes: assume 2× the throughput of B200 at similar power, with 1.5× HBM capacity. Real numbers when NVIDIA publishes official specs.

Cooling and power considerations

Datacenter readiness for modern GPUs is a real concern.

Power consumption progression

GPU	TDP	Recommended cooling
A100 80GB	400W	Air
H100 SXM	700W	Air (high-density)
H200 SXM	700W	Air
B100 SXM	700W	Air (with retrofits)
B200 SXM	1000W	Liquid recommended
GB200 (per GPU)	1200W (with NVLink)	Liquid required

The shift past 1000W per GPU is qualitative: existing datacenters often can't air-cool that density. Liquid cooling adds capital cost and operational complexity.

Rack density

A standard 42U rack with 8-GPU servers can hold 4-5 H100 nodes (32-40 GPUs). With B200 NVL72, a single rack holds 72 GPUs but draws 120 kW — beyond what most facilities can deliver per rack.

For production B200 deployment, plan:

Rack power: 60-120 kW per rack.
Liquid cooling distribution.
High-bandwidth networking aggregated at the rack level.

This is significant infrastructure. Many cloud providers handle this; on-prem deployments need facility upgrades.

Power efficiency (perf/watt)

Each generation improves perf/watt:

A100: 312 GFLOPs/W (FP16).
H100: 2.8 TFLOPs/W (FP16). ~9× A100.
B200: 4.5 TFLOPs/W (FP16). ~1.6× H100.
R100 (announced): expected ~9 TFLOPs/W.

For datacenters constrained by total power, perf/watt is the metric, not absolute compute. A 1MW facility with B200s can train ~2.5× the workload of the same facility with H100s.

Liquid cooling specifics

Two approaches:

Direct-to-chip (DLC): cold plate on the GPU; coolant flows directly. Used in DGX H100/B200 and HGX systems. Cools 1000W+ chips effectively.

Immersion cooling: the entire server submerged in dielectric fluid. Higher density than DLC but operationally unusual. Used in some specialized deployments.

Most production GPU clusters in 2026 use DLC. Immersion is niche.

Pricing and availability

Mid-2026 cloud pricing (rough, varies by provider and term):

GPU	Reserved (1yr)	On-demand	Spot
A100 80GB	$1.10/hr	$1.50/hr	$0.40/hr
H100 SXM	$2.00/hr	$4.00/hr	$1.30/hr
H200 SXM	$3.00/hr	$5.50/hr	$1.80/hr
B100	$3.50/hr	$6.50/hr	$2.50/hr
B200 SXM	$4.50/hr	$8.00/hr	$3.00/hr
GB200 NVL72	$7.00/hr per GPU	$11.00/hr	rare

Availability:

A100, H100: deep markets, easy to get.
H200: easier than B200, available through major clouds.
B200: tight supply through 2026. Expect 2-4 month lead times for new contracts.
GB200: very tight. Major clouds prioritize their own deployment over external customers.

Lease vs buy economics

For sustained workloads:

Lease (cloud):

Pros: zero capex, scale up/down freely, no ops burden.
Cons: 50-100% premium over wholesale cost. Vendor lock-in.

Buy (on-prem or co-location):

Pros: amortized cost is 30-50% lower.
Cons: significant capex, multi-year commitment, ops burden.

Crossover: at ~2 years sustained 24/7 utilization, buying breaks even. Beyond that, buying wins. Below, leasing usually wins.

For a startup growing fast: lease. Don't lock in capex when you might not need that hardware in 18 months.

For an established production deployment: buy. Or use long-term reserved cloud contracts (1-3 year terms) which approach buy economics.

Cloud GPU options compared

AWS:

P5 instances (H100): widely available, $4-6/hr/GPU on-demand.
P5e (H200): mid-2025 launch, premium pricing.
P6 (B200): Q1 2026 limited GA.
EFA networking. Solid but not class-leading.

GCP:

A3 (H100): widely available.
A3 Mega (H100 NVL): for memory-bound workloads.
A4 (B200): late-2025 launch.
TPU v5p as alternative.

Azure:

ND H100 v5: well-established.
ND H200 v5: good availability.
ND B200 v6: 2026 rollout.
InfiniBand on dedicated AI clusters.

CoreWeave:

H100 fleet: deep, often cheapest.
H200, B200: growing.
Reservation-friendly. Co-location-friendly.

Lambda:

H100, H200, B200.
Spot pricing competitive.
Smaller scale but flexible.

Oracle Cloud:

Aggressive H100/H200 pricing.
Bare-metal GPU instances.
Underrated option for cost-conscious deployments.

For most production workloads in 2026: AWS/GCP/Azure for ecosystem; CoreWeave/Lambda for cost; Oracle for specific large commits.

Workload fit: training vs inference

The two phases stress the hardware in opposite ways — if the distinction between training and inference is fuzzy, that explainer is worth reading first.

Training: Blackwell wins

For new training runs at scale:

B200 is 2-3× faster than H100 in real workload throughput.
GB200 NVL72 enables TP=72 for the largest models without going across InfiniBand.
FP4 training is emerging but not yet universal.

If you're starting a frontier-scale training run in 2026, you want B200 or GB200.

Inference: depends on context

Inference is bandwidth-bound for decode and capacity-bound for context. The decision tree:

Long context (32k+): H200 or B200. HBM capacity is the constraint.
High concurrency, moderate context: H200 or B200 for fewer replicas; H100 for cost-optimized scale.
Decode-heavy workloads: B200 (8 TB/s HBM crushes H100's 3 TB/s).
Cost-sensitive serving: H100 capacity is cheaper and widely available.

Inference on Blackwell with FP4

If you can tolerate FP4 quality cost, B200 with FP4 weights and FP4 KV is dramatically more capacity-efficient than H100 with FP8. Capacity per dollar can be 2-3× better.

For chat workloads (high quality tolerance), this is a win. For long-context retrieval (low quality tolerance), stick with FP8.

Picking the right SKU

Decision tree:

Are you training a new frontier model?
- Yes → B200 or GB200. Pick GB200 if you need TP > 8 within a single fabric.
Are you running inference?
- Long context (32k+)? → H200 (best capacity/$) or B200 (highest throughput).
- High concurrency moderate context? → H100 (cheapest scale) or H200.
- Cost-optimized? → H100 spot.
Are you doing research/dev?
- H100 or A100 spot. Lowest commitment, deepest market.
Are you doing edge deployment?
- Different question entirely. Apple Silicon, AMD, or smaller NVIDIA SKUs (L40S).

Capacity planning examples

Example 1: serving Llama-3 70B for 100 concurrent users at 32k context

H100 80GB: TP=2 needed. KV memory tight; 4-replica setup. 8× H100 minimum. ~$32/hr.
H200 141GB: TP=2, single replica handles ~180 concurrent at 32k. 2× H200 enough. ~$6/hr.
B200 192GB: TP=2, single replica handles ~250 concurrent at 32k. 2× B200 enough. ~$9/hr.

H200 wins cost-effectiveness for this workload.

Example 2: training a 7B from scratch in 14 days

H100 setup: 32 H100s, BF16 training. ~$1300/day × 14 = ~$18k.
B200 setup: 16 B200s (2× faster), FP8 training. ~$1700/day × 14 = ~$24k.

Wait — B200 costs more here? Yes, because the speed advantage doesn't outweigh the price premium for a 7B model. B200 wins for larger models where speed compounds.

Example 3: training a 405B model

H100 setup: 16,000 H100s for 90 days. ~$230M total.
B200 setup: 8,000 B200s for 60 days (2× faster + smaller cluster). ~$165M.

Now B200 wins on absolute cost despite higher hourly. The compounding speedup pays off at scale.

Capacity planning for inference clusters

Different workload mixes call for different SKU choices.

Chat workload (5k context, 100 concurrent users):

H100: 4-6 GPUs (TP=2, 2-3 replicas).
H200: 2-3 GPUs (TP=2, 1-2 replicas).
B200: 2 GPUs (TP=2, 1 replica).

For this profile, H200 is often most cost-effective. H100 is cheapest absolute; B200 is overkill.

Long-context RAG (32k context, 20 concurrent):

H100: 8 GPUs (TP=4, multiple replicas due to KV constraints).
H200: 4 GPUs (more KV per replica).
B200: 2 GPUs.

Longer context favors H200/B200's larger HBM. The math gets sharply better.

Reasoning models (long output, 50 concurrent):

H100: 16+ GPUs (decode-heavy, scaling out).
H200: 8+ GPUs (HBM bandwidth helps decode).
B200: 4-8 GPUs (FP4 compute + bandwidth).

Reasoning workloads are decode-bound; bandwidth-rich GPUs shine.

Capacity planning for training clusters

7B model fine-tuning (100 GPU-hours):

8× H100s for ~12 hours.
8× B200s for ~6 hours.

Doesn't matter much; small training is fast.

70B model fine-tuning (3000 GPU-hours):

32× H100s for ~4 days.
32× B200s for ~2 days.

B200 saves wall-clock time. If iteration speed matters, worth the premium.

405B pre-training (6.7M GPU-hours):

16,000× H100s for ~90 days.
8,000× B200s for ~60 days.

At this scale, B200 wins on absolute cost despite the price premium.

Workload-specific GPU selection deep dive

For each major AI workload, the right GPU differs.

LLM inference, single user

Latency-sensitive. ITL is dominated by HBM bandwidth (memory-bound decode).

Best: H200 or B200. Bandwidth premium directly translates to ITL improvement.
Acceptable: H100.
Avoid: A100 (lower bandwidth, slower decode).

LLM inference, multi-tenant chat

Throughput-sensitive. KV cache capacity determines concurrent users.

Best: H200 (capacity), or B200 (capacity + FP4 throughput).
Acceptable: H100 (cheap base).
Avoid: anything with <80GB HBM for 70B+ models.

LLM inference, long context

KV cache capacity is the binding constraint.

Best: H200 or B200. The 1.7-2.4× HBM jump matters.
Acceptable: H100 with TP=4+ (split KV across more GPUs).

LLM training, fine-tuning

Compute-bound. FP8 throughput matters.

Best: B200 (highest FP8/FP4 throughput).
Acceptable: H100/H200 (still excellent).

LLM training, frontier-scale pre-training

All metrics matter, hardware density wins.

Best: GB200 NVL72 (rack-scale TP).
Acceptable: standard 8-GPU DGX B200 or H100 nodes.

Image/video generation (diffusion models)

Compute-bound, high arithmetic intensity, modest HBM needs.

Best: B200 (FP4 throughput).
Acceptable: H100 (excellent in BF16/FP8).
Avoid: low-VRAM consumer GPUs for large models.

Embeddings / classification

Small models, batched inference.

Best: H100 (cost-optimized).
Overkill: H200, B200.
Reasonable alternative: L40S (smaller datacenter GPU).

Reinforcement learning training

Mix of compute (policy update) and data movement (rollout collection).

Best: H100/B200 with NVLink for distributed rollout.
Special considerations: high cluster-wide throughput more important than single-GPU peak.

Edge inference

Latency, power, cost. Datacenter GPUs don't apply.

Best: NVIDIA Jetson Orin, Apple Silicon, Qualcomm AI chips.
Alternative: small datacenter GPUs (L4, L40S) for edge servers.

Power and cooling: from air to liquid

The transition from H100 to B200 marks the transition from air-cooled to liquid-cooled datacenter design for AI. The implications go beyond the GPU itself.

Air cooling limits

A standard datacenter rack supports 15-50 kW of compute power with air cooling. H100 servers at 8 GPUs per node at 700W each plus CPU and other overhead are around 8 kW per node; 4-5 nodes per rack is feasible with air.

B200 servers at 1000W per GPU push thermal density higher. 8-GPU B200 nodes are around 12 kW. Per-rack feasibility with air drops to 3-4 nodes; less than half the GPUs per rack.

Liquid cooling enables density

Direct-to-chip liquid cooling enables 100+ kW per rack. GB200 NVL72 at 120 kW is a typical example. The density allows much more compute per square foot of datacenter floor space, which matters because frontier-scale training requires aggregated low-latency compute.

Cooling infrastructure cost

Retrofitting a datacenter for liquid cooling is a substantial investment — typically $5-10M for a single hall to add chilled-water loops, CDUs (Coolant Distribution Units), and the associated plumbing. Greenfield datacenter buildouts for AI now design liquid-cooling from the start.

Power supply implications

A 120 kW rack draws roughly 600 A at 200V three-phase. Most existing datacenters do not have 600A circuits available to individual racks. The infrastructure upgrades to support frontier AI compute include both cooling and power distribution.

What this means for cloud customers

When you rent a GB200 NVL72 in a cloud, you're paying for the infrastructure investment via the per-GPU hourly rate. The premium over H100 reflects both the GPU cost and the facility cost of supporting the higher-density compute.

Heat reuse and sustainability

Liquid cooling produces useful low-grade heat (60-80°C coolant temperature) that can theoretically be reused for district heating, water heating, or other applications. A few datacenter operators (Crusoe, some European operators) are deploying heat-reuse schemes. The economics are workload-dependent and geographically limited.

For deeper coverage of the rack-scale topology, see NVLink and rack-scale topology.

NCCL and GPU-to-GPU communication

The NVIDIA Collective Communications Library (NCCL) is the standard for GPU-to-GPU communication in AI workloads. Understanding NCCL at a high level helps in evaluating GPU SKU choices.

What NCCL does

NCCL implements collective communication primitives (all-reduce, all-gather, reduce-scatter, broadcast, all-to-all) optimized for NVIDIA GPUs. The library is the foundation of distributed training in PyTorch, DeepSpeed, Megatron-LM, and most other frameworks.

Topology-aware algorithms

NCCL detects the GPU topology (NVLink, NVSwitch, PCIe, InfiniBand, RoCE) and chooses appropriate algorithms. For 8-GPU nodes with NVLink, intra-node all-reduce runs in a ring or tree pattern at NVLink speeds. For multi-node, inter-node traffic uses RDMA over the network.

Performance per generation

A100 8-GPU NVLink: ~600 GB/s aggregate intra-node bandwidth.
H100 8-GPU NVLink: ~900 GB/s aggregate intra-node bandwidth.
B200 8-GPU NVLink: ~1800 GB/s aggregate intra-node bandwidth.
GB200 NVL72 NVSwitch: ~130 TB/s aggregate across the rack.

The pattern: each generation roughly doubles intra-node bandwidth; rack-scale (NVL72) is qualitatively different from per-node bandwidth.

Tuning matters

Default NCCL configurations work well in many cases but can be tuned for specific topologies. NCCL_IB_QPS_PER_CONNECTION, NCCL_IB_GID_INDEX, NCCL_NET_GDR_LEVEL, and other environment variables affect throughput. Frontier-scale training stacks invest significant engineering in NCCL tuning.

For deeper coverage of NCCL configuration and topology, see NCCL guide.

Alternative communication libraries

MSCCL (Microsoft): Extends NCCL with custom communication algorithms. Used in some Azure deployments.
RCCL (AMD): ROCm equivalent of NCCL for AMD GPUs.
Intel oneCCL: Intel's equivalent for Habana and other Intel accelerators.

For cross-vendor workloads, the differences in collective library performance matter. For NVIDIA-only deployments, NCCL is the standard.

Hopper-to-Blackwell migration tips

Practical considerations for teams migrating workloads from H100 to B200.

Software compatibility

CUDA code compiled for H100 (compute capability 9.0) runs on B200 (compute capability 10.0) via forward compatibility. New B200 features (FP4, TCGen5) require code recompiled with the newer CUDA toolkit.

PyTorch, TensorFlow, JAX all support B200 in their recent releases. The transition is typically painless at the framework level.

Performance differences

B200's improvements over H100:

2.25x more FP8 tensor compute.
2x more memory bandwidth (HBM3e at 8 TB/s vs HBM3 at 3.35 TB/s).
2x more aggregate NVLink bandwidth.
Native FP4 support.
2.4x more memory (192GB vs 80GB).

For most workloads, the wall-clock speedup over H100 is 1.5-2x. Workloads that benefit most: long-context inference (memory bandwidth helps), large-batch training (compute throughput helps), MoE serving (memory capacity helps).

Workloads that don't benefit much

Small-batch inference of small models: the additional compute and memory don't help; throughput-per-dollar may actually decrease at B200's higher price.

CPU-bottlenecked workloads: the GPU upgrade doesn't help if the bottleneck is elsewhere.

Tightly-coupled multi-rack training: B200's per-GPU improvements are partially negated by network bottlenecks that haven't proportionally improved.

Cost-effectiveness

B200 typically costs roughly 2x H100 hourly. For workloads achieving 1.5-2x throughput, the cost-per-token is similar to slightly better than H100. The crossover point depends on workload specifics.

For new deployments, B200 is increasingly the right choice. For existing H100 deployments, the migration depends on whether the workload benefits from B200's improvements enough to justify the capex.

Rubin family preview: R100, GR200, and rack-scale plans

Rubin is NVIDIA's post-Blackwell architecture, with first products expected in late 2026 and large-scale availability in 2027. Public information is partial; the publicly disclosed information at the time of writing follows.

R100

The Rubin-architecture flagship GPU. HBM4 memory at higher capacity per stack than HBM3e (early indications suggest 288GB per GPU as a starting point). Tensor-core throughput improvements roughly 1.5-2x B200 per public statements. Expected ship dates: late 2026 for initial customers, broader availability in 2027.

GR200

The Rubin-equivalent of GB200 — paired Rubin GPU and Grace CPU on a single board. Expected to be the building block of rack-scale Rubin systems.

Rubin NVL144 (planned)

A 144-GPU rack-scale system anticipated as the Rubin generation's flagship rack. Roughly twice the GPU count of NVL72 in a similar physical form factor. The interconnect generation (NVLink 6, presumably) supports the larger coherent compute domain.

What Rubin changes

The HBM4 memory bandwidth (anticipated 1.5-2x HBM3e) addresses the memory-bandwidth bottleneck that dominates LLM inference. The larger memory capacity per GPU eases the constraints on large-model serving without aggressive quantization.

For training, the throughput gain is similar in shape to Hopper-to-Blackwell — incremental rather than transformative. The structural pattern of frontier training (data parallelism, tensor parallelism, pipeline parallelism, FP8/FP4 precision) is unchanged.

Timeline expectations

NVIDIA's typical cycle: announce at a major event 12-18 months before ship, ship to limited customers initially, then broad availability 6-12 months later. Rubin announcement at GTC 2024 implies ship in late 2026 to early 2027, broad availability mid-2027 to late 2027.

For customers buying decisions in 2026, the question is whether to commit to Blackwell now or wait for Rubin. The 2026 answer: commit to Blackwell — the wait would be 12+ months and the productivity loss outweighs the per-GPU performance gap.

Secondhand pricing trajectory

GPU pricing in the secondary market is informative for both buyers and sellers, and reveals how depreciation actually works for AI hardware.

A100 secondhand market

A100 80GB SXM cards on the secondhand market in mid-2026 sell for roughly $7,000-$12,000 each (vs original list around $12,000-$15,000 in 2021-2022). The relatively shallow depreciation reflects continued utility — A100 is still useful for inference and fine-tuning workloads where H100 isn't necessary.

H100 secondhand market

H100 80GB SXM secondhand pricing in mid-2026: roughly $18,000-$25,000 each (vs original around $30,000-$40,000). Depreciation is slower than typical compute hardware because H100 supply remains constrained.

H100 PCIe and lower SKUs

The PCIe form factor of H100 secondhands at $13,000-$18,000. The lower compute and bandwidth (vs SXM) cap the secondhand value.

Pre-Hopper SKUs

V100 and earlier are largely irrelevant for new AI deployments. Secondhand prices are nominal ($500-$2000 for V100 32GB). These cards still find use in non-AI compute workloads.

When secondhand makes sense

For cost-driven deployments running stable production workloads (e.g., inference of established open-weight models), secondhand A100 or H100 cards are highly cost-effective. The implied dollar-per-FLOP is dramatically below new-card pricing.

For workloads where the latest generation matters (frontier training, latest features), buy new.

Liquidity and timing

Secondhand market liquidity is workload-driven. Major hyperscaler refresh cycles (typically every 3-4 years per generation) flood the secondhand market with older cards. The 2025-2026 period saw substantial H100 secondhand supply as cloud providers transitioned to H200 and B200.

For sellers: time sales to coincide with new-generation launches, when demand for upgrade trades is highest.

CUDA compatibility across generations

Each GPU generation has a compute capability number that determines CUDA compatibility. Workloads compiled for newer compute capabilities cannot run on older hardware.

Compute capability per generation

A100: 8.0
H100, H200: 9.0
B100, B200: 10.0
Rubin (expected): 11.0 or 12.0

What compute capability affects

Available instructions (tensor-core types, atomic operations, memory operations).
Maximum threads per block, register count, shared memory size.
Specific features (TMA on Hopper, FP4 on Blackwell, etc.).

Backward compatibility

CUDA binaries compiled for an older compute capability run on newer hardware (forward compatibility). The reverse is not true — code requiring Blackwell features won't run on H100.

For libraries and frameworks, this means each release typically supports a range of compute capabilities. Production code paths typically target 7.5 (T4-era) through 10.0 (Blackwell) to cover most deployment scenarios.

Driver and runtime versions

In addition to compute capability, the NVIDIA driver version and CUDA toolkit version matter. New GPU generations require minimum driver versions (B200 requires driver 555+ as of mid-2025); workloads relying on cutting-edge driver features may not run on older drivers.

Production stacks typically pin a specific CUDA toolkit version and driver minimum to manage compatibility. Mismatches between application requirements and deployed drivers are a common source of bugs.

Per-SKU deep dive: every datacenter GPU NVIDIA ships

The NVIDIA datacenter lineup has more SKUs than most teams realize. A reference for each.

A100 40GB SXM

The original Ampere datacenter GPU (2020). 40GB HBM2e, 1.6 TB/s bandwidth, 312 TF FP16 tensor compute. Largely retired from new production deployments in 2026 but still common in older fleets. Best for: inference of small-to-medium models, fine-tuning of 7-13B models, research.

A100 80GB SXM

The capacity-upgraded variant (2020). Same compute as A100 40GB, doubled memory at 80GB HBM2e, 2 TB/s bandwidth. The workhorse of 2021-2023 training. Still useful for inference of 70B-class models in BF16 or 405B in INT4. Best for: cost-sensitive inference, smaller training runs.

A100X (BlueField + A100)

A specialty SKU that combined the A100 with a BlueField DPU on a single card. Limited adoption; the use cases (network-accelerated AI) didn't materialize broadly. Mostly historical interest.

H100 SXM 80GB

The Hopper flagship (2022-2023). 80GB HBM3, 3.35 TB/s bandwidth, 1979 TF FP8 tensor compute. The standard for AI training and inference through 2024-2025. Best for: training and inference of any model that fits in 80GB, with FP8 support.

H100 PCIe 80GB

The PCIe form factor of H100. Slightly lower TDP (350W vs 700W SXM), lower memory bandwidth (2 TB/s), no NVLink. Useful for systems that can't accommodate SXM. The compute is somewhat reduced (around 70% of SXM throughput in practice). Best for: smaller deployments, retrofit into existing servers.

H100 NVL

A dual-card configuration where two H100 PCIe cards connect via NVLink bridge for shared memory. 188GB total HBM3 (94GB per card with some reserved). Targeted at LLM inference where the memory matters more than raw compute. Best for: inference of large models on smaller systems.

H200 SXM 141GB

The memory upgrade of H100 (2024). Same compute as H100 SXM (1979 TF FP8), but 141GB HBM3e at 4.8 TB/s bandwidth. The capacity and bandwidth upgrades matter for inference (KV cache, long context) more than training. Best for: long-context inference, large-model inference, training with memory-bound workloads.

H200 PCIe / H200 NVL

The PCIe and dual-card variants of H200. Same trade-offs as H100 PCIe and H100 NVL but with the larger memory. Best for: PCIe-form-factor deployments with large memory needs.

B100 SXM

The lower-end Blackwell SKU (early 2025). 192GB HBM3e, 8 TB/s bandwidth, 7000 TF FP8 tensor compute. Roughly 2x H100 performance. Lower TDP (700W) than B200, simpler cooling. Best for: existing H100-class deployments looking to upgrade without changing infrastructure.

B200 SXM

The Blackwell flagship (2025). 192GB HBM3e, 8 TB/s bandwidth, 9000 TF FP8 tensor compute, 18 PF FP4 tensor compute. 1000W TDP, requires liquid cooling in dense deployments. Best for: frontier training, large-model inference, new deployments.

GB200 NVL36 / NVL72

Rack-scale Blackwell configurations. NVL72 is 36x dual-GB200 boards (72 GPUs total) connected by NVSwitch in a single rack, presenting as a single logical compute domain. 30TB+ of aggregate HBM3e. NVL36 is the half-rack variant. These are the frontier-scale training systems of 2025-2026. Best for: 100B+ model training, large-scale inference of MoE models.

GB300 (planned late 2026)

The mid-cycle refresh of GB200. Higher memory (288GB per GPU expected), slightly higher compute, improved interconnect. Publicly announced; ship dates expected in late 2026.

RTX 6000 Ada Generation

A workstation-class GPU based on the Ada (RTX 4090) architecture but in datacenter form factor. 48GB GDDR6 ECC. Useful for visualization, smaller-model inference, and workstation deployments. Not competitive with H100 for serious AI training.

RTX Blackwell Pro 6000

The Blackwell-architecture professional card. 96GB GDDR7 ECC. Workstation form factor. Better than RTX 6000 Ada for AI workloads but still well behind datacenter SXM SKUs for serious work.

L4

The low-power Ada-architecture inference card. 24GB GDDR6, 72W TDP. Targeted at edge inference and high-throughput inference of smaller models. Best for: cost-optimized inference of 7B-class models in batch.

L40S

The higher-power L-series card. 48GB GDDR6, 350W TDP. Better than L4 for medium-scale inference. Best for: video AI workloads, mid-scale inference deployments.

T4

The previous-generation low-power inference card (Turing architecture, 2018). 16GB GDDR6, 70W TDP. Largely obsolete for AI workloads in 2026 but still widely deployed for legacy inference.

NVLink, NVSwitch, and PCIe generation history

Inter-GPU bandwidth has scaled faster than per-GPU compute over the last 5 years. The history matters because workloads designed for older interconnects don't always optimize for newer ones.

NVLink generations

NVLink 1 (2016, P100): 20 GB/s per link.
NVLink 2 (2017, V100): 25 GB/s per link.
NVLink 3 (2020, A100): 25 GB/s per link, more links per GPU (12).
NVLink 4 (2022, H100): 25 GB/s per link, 18 links per GPU, 900 GB/s aggregate.
NVLink 5 (2024, B200): 50 GB/s per link, 18 links per GPU, 1800 GB/s aggregate.

The pattern: link speed roughly doubles every 2-3 generations, and the number of links has steadily increased. Each generation enabled new rack-scale architectures.

NVSwitch generations

NVSwitch 1 (2018, DGX-2): 6.5 TB/s aggregate, 16 GPUs.
NVSwitch 2 (2020, DGX A100): 4.8 TB/s aggregate.
NVSwitch 3 (2022, DGX H100): 13.5 TB/s aggregate, 8 GPUs per system.
NVSwitch 4 (2024, GB200): 130 TB/s aggregate at rack scale (NVL72), 72 GPUs.

The dramatic jump at NVSwitch 4 is what enabled rack-scale training as a coherent compute domain.

PCIe generations

PCIe matters for CPU-to-GPU communication, GPU-to-storage, and GPU-to-network on PCIe-attached cards.

PCIe Gen 3 (16x): 16 GB/s.
PCIe Gen 4 (16x): 32 GB/s. Standard through Ampere.
PCIe Gen 5 (16x): 64 GB/s. Standard on H100 and later.
PCIe Gen 6 (16x): 128 GB/s. Begins shipping on Blackwell systems and beyond.

PCIe Gen 5 is the practical floor for H100-class deployments; Gen 6 is recommended for Blackwell.

Cross-references

For deeper coverage of how these interconnects matter at rack scale, see NVLink and rack-scale topology. For the networking layer between racks, see AI training networking.

Multi-Instance GPU (MIG) in detail

MIG (Multi-Instance GPU) partitions a single physical GPU into multiple isolated logical GPUs, each with its own memory, compute slices, and L2 cache. Introduced on A100, refined on H100, supported on H200 and B200.

What MIG actually does

A H100 80GB can be partitioned into up to 7 MIG instances. Each instance has dedicated memory (the smallest size is 10GB, the largest is the full 80GB), compute resources (SM slices), and memory bandwidth. Workloads in different MIG instances cannot interfere with each other (hardware isolation).

The use case: multi-tenant inference serving where workloads are too small to use a full H100 individually but too security-sensitive to share memory with other workloads.

MIG profiles

Standard H100 MIG configurations:

1g.10gb: 1/7 of compute, 10GB memory. Smallest unit.
2g.20gb: 2/7 of compute, 20GB memory.
3g.40gb: 3/7 of compute, 40GB memory.
7g.80gb: full GPU.

Combinations: 7x 1g.10gb, 3x 2g.20gb + 1x 1g.10gb, etc. The total compute slices must sum to 7; the total memory to 80GB.

When MIG helps

Multi-tenant serving of small models (where each tenant gets a slice).
Mixing workload types on a single GPU (one slice for training, others for inference).
Compliance scenarios that require hardware-isolated tenants.

When MIG hurts

Large-model workloads that need the full GPU. MIG partitions waste capability.
Training workloads, which typically need NVLink and multi-GPU communication that MIG complicates.
Latency-sensitive serving, since each MIG slice has lower throughput than the full GPU.

For multi-tenant inference patterns that use MIG, see multi-tenant LoRA serving.

Tensor Memory Accelerator and TCGen5

Hopper introduced the Tensor Memory Accelerator (TMA); Blackwell extended tensor-core capabilities with TCGen5. These are the hardware features that enable the precision and throughput of modern training and inference.

TMA on Hopper

The TMA is a dedicated hardware unit for asynchronous data transfer between global memory (HBM) and shared memory (the SM-level cache). On older architectures, programmers had to use cooperative thread programming to move data from HBM to shared memory before tensor-core operations. TMA does this asynchronously with simpler programming.

The practical impact: high-throughput attention and matmul kernels became much easier to write. FlashAttention and similar kernels benefit substantially from TMA.

Tensor cores on Hopper

H100's tensor cores added FP8 support (both E4M3 and E5M2 formats), with throughput 4x BF16. They also added "asynchronous wgmma" — warp-group matmul instructions that issue matmul operations asynchronously and complete in the background while other operations proceed.

TCGen5 on Blackwell

B200's tensor cores ("TCGen5") add native FP4 support (both NVFP4 and MXFP4 formats), with throughput 2x FP8. They also support new operand sizes and improved scheduling, increasing effective utilization on transformer workloads.

The Blackwell tensor cores also include "tensor memory" — a small dedicated cache for tensor operands that reduces the bandwidth pressure on shared memory and L2.

Precision-format support matrix

Generation	FP32	TF32	BF16	FP16	FP8 E4M3	FP8 E5M2	FP4 NVFP4	FP4 MXFP4
A100 (Ampere)	yes	yes	yes	yes	no	no	no	no
H100 (Hopper)	yes	yes	yes	yes	yes	yes	no	no
H200 (Hopper)	yes	yes	yes	yes	yes	yes	no	no
B100/B200 (Blackwell)	yes	yes	yes	yes	yes	yes	yes	yes
Rubin (planned)	yes	yes	yes	yes	yes	yes	yes	yes

The pattern: each generation adds support for the next lower precision while retaining all higher precisions. FP4 is the current frontier; what comes after FP4 (FP2? log-quantized formats?) is research-stage.

Multi-vendor comparison: AMD, TPU, Trainium, others

NVIDIA dominates AI compute, but the multi-vendor landscape matters for cost, availability, and strategic optionality.

AMD MI300X, MI325X, MI355X

AMD's MI300X (2024) competes with H100. 192GB HBM3 (more than H100's 80GB), 5.3 TB/s bandwidth, comparable tensor-core throughput. The software stack (ROCm + flash attention + vLLM AMD backend) has matured significantly through 2025; production AMD inference is competitive in 2026 for many workloads.

MI325X (mid-2024) is the memory upgrade: 256GB HBM3e, 6 TB/s bandwidth. MI355X (planned for 2025-2026) is the next-generation with reported compute improvements approaching B200 levels.

The economic argument for AMD: typically 20-40% cheaper than equivalent NVIDIA SKUs, with comparable performance for inference. Training is more challenging because the software stack lags CUDA. Production AMD training is feasible but less mature.

Google TPU v5, v6 (Trillium), Ironwood

TPUs are Google's internal AI accelerators, also available externally through Google Cloud. TPU v5p (2024) competes with H100; TPU v6 Trillium (2024) competes with B200. Ironwood (2025) is the latest generation.

TPUs use a different memory model (no HBM, custom on-chip memory hierarchy), different precision support (Google's BF16-derivative formats), and different software (JAX/Flax with XLA, optional PyTorch via PJRT). For Google-internal workloads and customers using JAX, TPUs are highly competitive. For PyTorch-based stacks, the integration cost is real.

AWS Trainium, Inferentia

AWS's custom AI silicon. Trainium2 (2024) for training; Inferentia2 for inference. Pricing is significantly cheaper than NVIDIA on AWS but the software stack (Neuron SDK) requires specific integration. Best for AWS-native workloads where the cost matters and the software work is acceptable.

Cerebras WSE-3

The wafer-scale chip. 900,000 cores on a single wafer, 44GB on-chip SRAM, optimized for very large model training in a single chip. Different programming model entirely. Niche but useful for specific frontier workloads.

Groq LPU

A latency-focused inference chip. 230 MB of on-chip SRAM, no HBM. Optimized for extremely low-latency inference of small-to-medium models. Production deployments are growing for specific use cases (real-time agents, voice interfaces).

Tenstorrent Wormhole, Blackhole

Tenstorrent's accelerators. Distinct programming model (Tensix cores with explicit data movement). Early adoption; software stack still maturing.

SambaNova SN40L

A reconfigurable dataflow architecture for AI. Niche; specific customers, specific workloads.

Vendor comparison summary

Vendor	Best for	Software stack	Production maturity (2026)
NVIDIA H100/B200	Everything	CUDA, mature	Very high
AMD MI300X/MI325X	Inference, cost-sensitive	ROCm, maturing	High for inference
Google TPU v5/v6	JAX workloads, GCP customers	XLA, JAX, mature for Google	High within ecosystem
AWS Trainium/Inferentia	AWS-native workloads	Neuron SDK	Moderate
Cerebras WSE-3	Frontier-scale training	Custom	Niche
Groq LPU	Ultra-low-latency inference	Custom	Growing
Tenstorrent	Research, exploration	TT-NN, growing	Early
SambaNova	Reconfigurable workloads	Custom	Niche

The NVIDIA dominance is real but not absolute. For specific use cases (inference cost, ultra-low latency, GCP-native workloads), alternative vendors are competitive.

GB200 NVL72: rack-scale engineering

The GB200 NVL72 rack is the most architecturally distinct hardware NVIDIA has released. Understanding it clarifies where frontier AI infrastructure is heading.

What's in the rack

72 B200 GPUs and 36 Grace CPUs across 18 compute trays. NVSwitch fabric connects all 72 GPUs into a single logical compute domain via NVLink 5. Aggregate HBM3e: roughly 13.8TB across the rack. Aggregate FP8 compute: roughly 650 PFLOPS.

The rack also includes the NVSwitch trays (separate from the compute trays), power distribution, and liquid cooling infrastructure. Total rack TDP: approximately 120 kW — far above the typical 30-50 kW that traditional datacenter racks support.

Liquid cooling

Air cooling is infeasible at 120 kW. The rack uses direct-to-chip liquid cooling, with cold plates on the GPUs and CPUs and a closed-loop coolant system. Datacenter operators deploying NVL72 must upgrade their facility to support liquid cooling, which is a substantial infrastructure investment.

Power distribution

The rack draws roughly 120 kW. Datacenter operators typically dedicate one or two 200A 3-phase circuits per rack for the NVL72. The associated PDU (Power Distribution Unit) is custom NVIDIA hardware integrated into the rack design.

Cabling

The NVSwitch fabric requires roughly 5000 individual copper cables between the compute trays and the switch trays. NVIDIA pre-cables these at the factory; field service involves swapping entire trays rather than reseating individual cables.

What the rack enables

The 72-GPU coherent compute domain means workloads that would have required multi-rack tensor-parallel communication on previous generations can now run within a single rack with NVLink bandwidth. The throughput gain for parallelism-bound workloads is substantial — frontier training MFU improvements of 20-40% are reported on NVL72 vs equivalent H100 setups.

What it costs

A GB200 NVL72 rack costs in the $3-5M range as of 2026, including the infrastructure upgrades for liquid cooling and power. The depreciation period is 4-5 years; effective hourly cost per GPU runs around $1.50-2.50/hr fully amortized.

For deeper coverage of rack-scale topology, see NVLink and rack-scale topology.

Export controls: H800, H20, B30 China-market variants

US export controls on advanced GPUs have evolved through multiple revisions since 2022. The current landscape has specific China-market variants and explicit performance caps.

H800 (2023-2024)

The original China-market H100 variant. Reduced NVLink bandwidth (400 GB/s instead of 900 GB/s) and reduced FP64 performance. FP8 and BF16 compute were unchanged from H100. Widely used in China for AI training before further export-control tightening.

H20 (2024-2025)

The post-tightening China variant of H100. Substantial reductions: FP8 compute reduced by approximately 50% from H100, total system performance approximately 40% of H100. The H20 was designed specifically to fit under the revised export control thresholds.

Adoption in China was significant but mixed — the H20's compute reduction made it less attractive than alternatives like the AMD MI300X (also subject to export controls but with different specifics) or domestic Chinese accelerators.

B30 (planned)

The China-market variant of B200, planned for 2026. Performance specifics are subject to changing US export controls; the design is reportedly tuned to fit just under the latest performance caps.

What the controls actually restrict

The current US export controls (as of 2026) cap total system performance and aggregate interconnect bandwidth above specific thresholds. The thresholds are revised periodically; the trend has been tightening through 2022-2026.

The result: top-tier NVIDIA hardware (B200, GB200 NVL72) cannot be exported to China or many other restricted jurisdictions. Cut-down variants exist but are increasingly limited.

Implications for global AI capacity

China has invested heavily in domestic alternatives (Huawei Ascend 910B, 910C; alternative GPU vendors). The 2026 landscape is fragmented: NVIDIA-dominant in the US/EU, mixed in China with growing domestic share, varying in other jurisdictions.

For builders, the practical implication: hardware roadmaps differ by region, and the specific GPUs available depend on jurisdiction. Production deployments serving global users may need to accommodate multiple hardware platforms.

Cloud availability matrix: AWS, Azure, GCP, specialist

What SKUs each major cloud actually offers, in 2026.

AWS

p4d (A100 40GB): generally available, retiring.
p4de (A100 80GB): generally available.
p5 (H100 80GB): generally available, large capacity.
p5e (H200 141GB): generally available, growing capacity.
p5en (H200 NVL): limited availability.
p6 (B200): rolling out through 2025-2026, capacity-constrained.
Inferentia2, Trainium2: generally available, AWS-specific software.

Azure

ND A100 v4: generally available.
ND H100 v5: generally available, large capacity.
ND H200 v5: generally available.
ND B200 v5: rolling out late 2025-2026.

GCP

A2 (A100): generally available.
A3 (H100): generally available.
A3 Ultra (H200): generally available.
A4 (B200): rolling out 2025-2026.
TPU v5p: generally available.
TPU v6 Trillium: generally available.
TPU Ironwood: rolling out 2025-2026.

Specialist clouds

CoreWeave: H100, H200, B200 — all generally available.
Lambda: H100, H200, B200 — all generally available.
Crusoe: H100, H200, limited B200.
RunPod: H100, H200, B200 (community tier), various consumer GPUs.

What this means for capacity planning

For most production workloads in 2026, H100 and H200 are widely available across providers; pricing varies by 30-50% depending on commitment level. B200 supply is constrained — most providers have waitlists or limited allocations through mid-2026. The supply situation typically improves 12-18 months after a SKU's launch.

For background on the cost economics across providers, see decentralized GPU compute and AI inference cost economics.

Workload-to-SKU map: what to use for what

A concrete reference matching workloads to recommended GPUs in 2026.

Frontier training (70B+ dense, 400B+ MoE)

Primary: B200, GB200 NVL72, or H200 cluster with strong interconnect.
Acceptable: H100 cluster (slower, more expensive per token).
Avoid: anything older than H100 — the FP8 support matters.

Mid-scale training (7-70B)

Primary: H100 SXM, H200 SXM.
Acceptable: A100 80GB, B200 (overkill but works).
Avoid: A100 40GB (memory pressure), L-series (no NVLink).

Fine-tuning (7-70B SFT or DPO)

Primary: H100 SXM, H200 SXM. Single node is fine.
Acceptable: A100 80GB, RTX 6000 Ada / Blackwell Pro 6000 for 7-13B.
Avoid: T4, L4 (too small).

Large-model inference (70B+ chat, frontier-class)

Primary: H200 SXM, B200, H100 NVL.
Acceptable: H100 SXM with aggressive quantization.
Avoid: anything without enough HBM for the model + KV cache.

Long-context inference (32K+ context)

Primary: H200, B200 (the HBM matters).
Acceptable: H100 with shorter context or aggressive quantization.
Avoid: A100 80GB unless context is moderate.

Small-model inference (7-13B)

Primary: L40S, L4, H100 (any), consumer GPUs (4090, 5090).
Most cost-effective: L4 for batch inference, L40S for interactive.

Embedding generation, batch inference

Primary: L4, L40S, A100 (any).
Cost-optimized: any of the above with high utilization. Consumer GPUs on decentralized networks also fit.

Multimodal (vision-language, audio)

Primary: H100, H200, B200 (vision tokens add memory pressure).
See multimodal serving for serving-side considerations.

Reasoning workloads (test-time compute)

Primary: H200 (large KV cache), B200.
See reasoning model serving.

The bottom line

The named problem is the arithmetic-intensity wall: GPUs accumulate flops far faster than HBM can deliver bytes, so most AI kernels — especially LLM decode — sit memory-bound while expensive tensor cores idle. NVIDIA's answer across Hopper, Blackwell, and Rubin is to raise both the memory roof (HBM capacity and bandwidth) and the compute roof (FP8, FP4) in lockstep, then glue GPUs together with NVLink so a node behaves like one big device. The single biggest lever in SKU selection is matching HBM bandwidth and capacity to your workload's flops-per-byte, not chasing peak TFLOPs.

What to do if you take only this away:

Compute your workload's arithmetic intensity before specifying hardware. Decode-heavy inference is memory-bound; pretraining is compute-bound; fine-tuning is in between.
For long-context inference, H200 often beats H100 by ~1.7× on the same compute — buy bandwidth, not flops.
For new training runs, B200 / GB200 NVL72 wins on perf-per-watt and on collective bandwidth for tensor-parallel sharding.
Keep H100 fleets for production until depreciation runs out — the spot market is deepest there, and the software stack is mature.
Treat FP4 as an inference precision in 2026; training in FP4 is still research.

Next, read collective communication for AI training for how NVLink and IB actually behave under load, and FP8 training tradeoffs for what those lower-precision tensor cores cost in numerical stability.

FAQ

Q: Should I buy H100 or H200 for inference?

H200 if your workload is long-context-heavy (32k+) or memory-bandwidth-bound. H100 if it's compute-bound or moderate-context.

Q: When does Blackwell make sense?

For new frontier training runs and for the highest-throughput inference workloads. Lower priority for cost-optimized inference and research.

Q: What about A100? Still useful in 2026?

Yes, in cost-optimized inference deployments where H100/H200 don't justify the price. A100s are still the cheapest GPU capacity, especially on spot.

Q: GB200 vs 9× DGX H100 — which is better for frontier training?

GB200 wins for very large models requiring TP > 8. Otherwise comparable; 9× DGX H100 is more flexible (can split into 8-GPU islands per workload).

Q: Will Rubin be a big jump?

Likely 2× perf-per-watt over Blackwell, similar magnitude to Hopper → Blackwell. Plan for 12-18 months between announcement and broad availability.

Q: Should I worry about hardware lock-in?

Somewhat. NVIDIA's CUDA ecosystem is dominant; AMD MI300/MI350 are catching up but not yet at parity for training. Most teams stay NVIDIA. Some are testing AMD as a hedge.

Q: What about Google TPUs?

If you're on GCP and using JAX, TPUs are competitive. Outside that ecosystem, NVIDIA dominates.

Q: What's NVLink vs InfiniBand?

NVLink connects GPUs within a node (or rack for GB200). InfiniBand connects nodes. NVLink is 10–20× faster than InfiniBand, but doesn't span nodes (except GB200).

Q: Can I mix H100 and H200 in the same cluster?

Yes, but TP/PP groups need same-GPU per group. You can have H100 replicas and H200 replicas serving different traffic.

Q: How long until B200 is widely available?

Tight through 2026. Major clouds prioritize their own use. Late 2026 / early 2027 should see broader availability.

Q: What about consumer GPUs (4090, 5090)?

Useful for development and small-scale serving. RTX 4090 has 24 GB, 5090 has 32 GB. Not competitive at scale due to limited NVLink and capacity, but fine for solo work.

Q: How does AMD MI300/MI355 compare?

AMD's MI300X has 192 GB HBM at 5.3 TB/s — competitive with B100 on capacity, lower on FP4 throughput. ROCm software ecosystem improved significantly in 2024-2025; vLLM, SGLang, PyTorch all work well.

For LLM inference, MI300X is a real alternative to H100/H200. For training, NVIDIA's ecosystem maturity (Megatron, NeMo) still gives an edge.

Pricing: MI300X is often 30-50% cheaper than H100 at similar performance levels.

Q: TPU v5p / v6 — competitive?

For Google Cloud customers using JAX, yes. TPUs are competitive with NVIDIA for LLM training when the workload fits TPU-friendly patterns.

Outside the JAX ecosystem, NVIDIA dominates. Migration cost from PyTorch/CUDA to JAX/TPU is substantial.

Q: Cerebras CS-3 — when does it win?

For specific workloads where its wafer-scale architecture shines (e.g., high-batch inference of medium-sized models), CS-3 can be competitive. Mainstream training is still NVIDIA-dominated.

Q: What's coming in Rubin (R100)?

Officially announced: ~2× perf-per-watt vs Blackwell, HBM4, NVLink-6, native FP4 + new lower-precision formats. Expected late 2026 / early 2027.

Plan around it: don't buy more Blackwell than you need, but don't wait if you have urgent capacity needs.

Q: How do I evaluate a GPU SKU?

Real workload throughput is the only metric that matters. Benchmark tokens/sec on your actual workload, not synthetic GEMM throughput.

Q: What about used H100s on the secondary market?

Available, often 30-50% off new pricing. Caveats: no NVIDIA support, possible thermal/wear issues, may have been mining cryptocurrency. For experimentation, fine. For production, prefer new or refurbished-with-warranty.

Q: When should I buy vs lease?

Sustained 24/7 utilization for 2+ years: buying wins. Bursty or growing workloads: lease.

Q: What's the upgrade path from A100 to H100?

Most workloads see 2-3× throughput improvement. Software is largely compatible (CUDA forward-compat). Tensor Engine for FP8 is the new piece to integrate.

For ROI, H100 typically pays back vs A100 within 6-9 months on heavy workloads.

Q: Should I worry about supply chain for GPUs?

Yes, for B200 in 2026. NVIDIA prioritizes hyperscalers; other customers face long lead times. Pre-order with multiple vendors; consider H100 as a fallback.

Q: How does NVLink-5 differ from NVLink-4?

2× bandwidth (1.8 TB/s vs 900 GB/s). New protocol features for SHARP-style in-network reductions. Backward-compatible with NVLink-4 in mixed clusters (limited to NVLink-4 speed).

Q: Why did NVIDIA release H200 alongside H100?

H100 was bandwidth-constrained for large-context LLM inference. H200's 4.8 TB/s HBM (vs H100's 3.0) addressed the bottleneck without needing a new chip generation. It's a memory refresh, not a new architecture.

Q: Is FP4 production-ready?

For inference: yes, becoming standard on Blackwell. For training: emerging, used in some research/frontier setups but not yet universal.

Q: How does GB200 differ from individual B200s?

GB200 is a tightly-integrated rack: 72 B200 GPUs with NVSwitch fabric for full all-to-all NVLink. Operates as one logical NVLink domain. Required for very large TP groups (TP=72) or specific MoE setups (EP=64+).

For most workloads, individual 8-GPU B200 nodes are sufficient. GB200 NVL72 is for frontier-scale training where rack-scale parallelism is needed.

Q: Why is Hopper still so widely used in 2026?

Three reasons: (1) supply (B200 is tight), (2) cost (H100 is cheaper), (3) operational maturity (every stack is well-tuned for Hopper). For most production workloads, H100 is the safe, cost-effective choice.

Q: Should I choose by manufacturer (NVIDIA SXM vs HGX OEM)?

DGX (NVIDIA-built): premium, NVIDIA support included. HGX (OEM-built like Dell, Supermicro, HPE): same compute, often 20-30% cheaper, OEM support.

For most production: HGX is fine. For "I want NVIDIA's blessing": DGX.

Q: How do I migrate workloads to Blackwell?

Code-wise, mostly drop-in. CUDA forward-compat. The big considerations:

FP8 → FP4 if you want the throughput gain. Requires calibration testing.
Validate stack version supports B200 (FlashAttention-3, vLLM 0.7+, etc.).
Power/cooling readiness on the deployment target.

Q: What's a good performance baseline on H100?

For LLM inference at typical settings:

Llama-3 70B FP8: ~30-40 tokens/sec/request single stream, ~1500-2500 tok/sec aggregate per 2× H100.
Llama-3 8B FP8: ~150 tok/sec/request single stream, ~5000+ aggregate.

If your achieved rate is much lower, profile.

Q: How do I check GPU health?

nvidia-smi  # current utilization, errors
nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,memory.used,power.draw --format=csv

For deeper diagnostics: NVIDIA's DCGM (Data Center GPU Manager).

Q: What's the difference between consumer and datacenter GPUs?

Consumer (RTX 4090/5090):

Cheaper.
Smaller HBM (24-32 GB).
Less NVLink (limited or absent in newer consumers).
Game-optimized clock speeds.

Datacenter (H100/B200):

Larger HBM (80-192 GB).
Full NVLink fabric for multi-GPU.
ECC memory.
Optimized for sustained load.

For production AI: datacenter. Consumer for development only.

Q: Can I do model parallelism on a single GPU?

If the model fits, no need. If it doesn't, you need multiple GPUs.

There's a niche: "expert offloading" where some layers offload to CPU during forward. Only useful for inference of models that barely fit.

Q: GPU specs I should always check before deploying?

HBM capacity (must fit weights + KV).
HBM bandwidth (determines decode throughput).
Compute throughput (FP8/BF16/FP4 TFLOPs).
NVLink topology (within-node).
PCIe topology (host-to-GPU).
Power and thermal limits.

Vendor spec sheets cover these. Check them before commitment.

Q: How does Hopper FP8 differ from Blackwell FP8?

Same numerics. Blackwell has higher absolute throughput per GPU (~2× Hopper) but the format and quality characteristics are identical.

Code that works on H100 FP8 works on B200 FP8. Just faster.

Q: How long do GPUs last in production?

Datacenter GPUs are designed for 5+ years sustained operation. In practice, deployments retire GPUs after 2-4 years due to obsolescence (newer generations are cheaper per FLOP).

Consumer GPUs in datacenter use have shorter useful life (2-3 years).

Q: How do I deal with GPU shortages?

Reserve capacity in advance with multiple vendors. Use spot/preemptible for buffer. Plan workload mix to use multiple GPU types.

For B200 specifically (tight in 2026): pre-order with multiple cloud providers. Have H100 fallback.

Q: What about edge AI hardware?

Different game. NVIDIA Jetson Orin (datacenter capabilities at 60W), Apple Silicon for development, Qualcomm AI for mobile.

Not interchangeable with datacenter H100/B200. Different deployment patterns.

Q: Should I use cloud or on-prem?

Cloud for variable workloads, on-prem for predictable 24/7 utilization >2 years.

Math: cloud premium is ~50-100% over wholesale. On-prem amortizes to wholesale + ops cost over time.

Below ~1.5 years sustained: cloud. Above: lean toward on-prem.

Q: How does CoreWeave compare to AWS?

CoreWeave: GPU-specialist, often 30-40% cheaper than AWS for equivalent SKUs. Smaller scale, less integrated services.

AWS: full ecosystem, broader integrations, higher pricing.

For pure GPU compute: CoreWeave often wins. For "AI as part of a larger AWS deployment": AWS.

Q: What's the difference between an H100 SXM and an H100 PCIe?

The SXM form factor connects to NVLink and is used in 8-GPU servers like DGX H100 and HGX H100. The PCIe form factor connects via PCIe (no NVLink) and is used in standard servers. SXM has higher TDP (700W vs 350W) and higher memory bandwidth (3.35 TB/s vs 2 TB/s). For multi-GPU training, SXM is strictly better; for single-GPU inference, PCIe is acceptable.

Q: How does the H100 NVL differ from a regular H100?

H100 NVL is a dual-card configuration where two H100 PCIe cards connect via an NVLink bridge for shared 188GB memory. Designed for LLM inference where memory matters more than raw compute. Less common than standard H100 deployments; useful for serving large models on PCIe-only systems.

Q: What's MIG and when does it help?

Multi-Instance GPU partitions a single GPU into up to 7 isolated logical GPUs, each with dedicated memory, compute, and bandwidth. Useful for multi-tenant inference of small models where each tenant needs hardware isolation. Not useful for training or single-tenant large-model workloads.

Q: Should I use FP4 inference in 2026?

For Blackwell-based deployments: yes, increasingly. FP4 NVFP4 and MXFP4 inference gives 2x throughput vs FP8 with minor quality cost. For Hopper deployments: not available — H100 and H200 don't have native FP4 support. The tooling (vLLM, SGLang, TensorRT-LLM) has matured through 2025-2026 for production FP4 inference.

Q: What's the TGI advantage with NVL72?

The 72-GPU coherent compute domain enables tensor parallelism and pipeline parallelism within a single rack at NVLink bandwidth. Workloads that previously required multi-rack communication (with InfiniBand bottlenecks) now stay within the rack. Reported MFU improvements of 20-40% for frontier training on NVL72 vs equivalent H100 setups.

Q: How does AMD MI300X compare on memory-bound workloads?

MI300X has 192GB HBM3 (vs H100's 80GB). For memory-bound inference workloads — long context, large batch size, MoE with many experts — the memory advantage matters. Production benchmarks show MI300X competitive with H100 on inference throughput per dollar; sometimes better, sometimes worse, depending on the workload and tuning.

Q: When does TPU make economic sense over H100?

For workloads running on Google Cloud already, TPU pricing is typically 20-30% cheaper than H100 equivalent at the same compute level. The integration cost is real — JAX is the natural fit; PyTorch via PJRT works but is less mature. For GCP-native workloads, TPU is often the right answer; for portability and ecosystem reasons, H100 still wins for most non-Google teams.

Q: What's a reasonable depreciation period for AI GPUs?

For H100, plan a 4-5 year useful life. The hardware will still work after 4-5 years; whether it's economically competitive depends on the rate of generational improvement. Hopper-to-Blackwell brought 2x performance; Blackwell-to-Rubin is expected similar. After 4 years, an H100 in 2026 will be 2 generations behind and likely uneconomic for new workloads.

Q: Should I buy or lease GPUs?

For predictable steady-state workloads at 100+ GPU scale, buying via colocation can be cheaper after 18-30 months than equivalent reserved cloud pricing. Below 100 GPU scale, leasing is usually better (the capital tie-up isn't worth the operational complexity). The crossover depends heavily on the team's existing infrastructure and ops capacity.

Q: How does power consumption affect TCO?

H100 SXM draws 700W under load. At $0.10/kWh, that's $613/year per GPU just for power. For 8-GPU servers, ~$5K/year per server. Add cooling (typically 30-50% of compute power), and total power TCO is around $7-10K/year per server. For B200 at 1000W per GPU, scale accordingly.

Q: What's the difference between HBM3 and HBM3e?

HBM3e has higher bandwidth (4.8 TB/s on H200 vs 3.35 TB/s on H100, same number of stacks). Same fundamental architecture, faster speed grade. The bandwidth improvement matters more for inference than training; for training the compute is usually the bottleneck.

Q: Will fewer NVIDIA generations come out faster?

NVIDIA's announced cadence is roughly one major generation every 2 years (Hopper 2022, Blackwell 2024, Rubin 2026). Mid-generation refreshes (H100 → H200, B200 → GB300) come faster. The pace has accelerated in 2024-2026; expect 12-18 months between major architectural generations going forward.

Q: What does the export-controlled "B30" mean for buyers?

B30 is NVIDIA's planned China-market variant of B200, designed to fit under US export-control performance caps. Specifications are reduced versions of B200. For buyers in China, B30 is the available variant; for buyers elsewhere, the full B200 is available. Export controls don't directly affect customers in compliant jurisdictions.

Q: Are Grace CPUs useful outside of GB200/GR200 systems?

The Grace CPU is ARM-based with high memory bandwidth (LPDDR5X). Mostly useful in the context of Grace-Hopper or Grace-Blackwell paired systems where the CPU-GPU memory architecture matters. Standalone Grace CPUs are available but less common; for x86-native workloads, traditional Intel/AMD CPUs typically win.

Q: Should I deploy on the latest CUDA toolkit or pin a specific version?

For production, pin a specific CUDA toolkit version and update it deliberately. New CUDA versions occasionally break framework compatibility or introduce subtle perf regressions. The 2026 production default is CUDA 12.4 or 12.6, with most major frameworks supporting both. Test new versions on a non-production stack before rolling out.

Hardware roadmap and what to expect

NVIDIA's announced roadmap and likely directions.

Rubin (R100) — late 2026 / early 2027

~2× perf-per-watt vs Blackwell.
HBM4 (~1 TB/s/stack).
NVLink-6 (~3.6 TB/s/GPU).
Native FP4 + new MXFP6 / FP6 formats.
Co-packaged optics (CPO) for some configurations.

Expect: 2-3× LLM training throughput vs B200. Capacity premium initially.

Rubin Ultra — 2027 / 2028

Successor to Rubin. Expected ~50% faster than R100. Specifications not yet public.

Beyond Rubin — 2028+

Speculative. Likely directions:

More aggressive optical interconnect.
Photonic computing for some operations.
Dedicated AI accelerator silicon (less general-purpose than GPUs).
3D-stacked memory beyond HBM4.

Don't plan around speculative future hardware. Plan around what's available within 18 months.

What this means for buyers

Don't over-buy Blackwell expecting it to last 5+ years; Rubin will displace it.
Plan 2-3 year refresh cycles.
Build software that ports forward (CUDA generally maintains forward compatibility).
Reserve capacity early; supply is consistently tight at the leading edge.

Practical procurement playbook

Buying GPUs at scale in 2026.

For 8-32 GPUs (small/medium scale)

Cloud: AWS, GCP, Azure, or specialty (CoreWeave, Lambda).
On-demand or 1-year reserved.
No need for direct procurement.

For 64-256 GPUs (medium scale)

Mix of cloud reserved + on-prem.
Direct relationship with NVIDIA partners (Dell, HPE, Supermicro).
Plan 6-12 months ahead.

For 1000+ GPUs (large scale)

Direct NVIDIA partnership.
Custom orders, reference architectures (DGX SuperPOD).
12-18 month lead times for new builds.
Significant infrastructure planning (power, cooling, networking).

For 10,000+ GPUs (frontier scale)

Multi-year strategic partnerships with NVIDIA.
Co-development of reference architectures.
Datacenter co-location or build-out.
18-36 month planning horizons.

This last tier is mostly hyperscalers and frontier labs. Most teams operate at scales below.

Key procurement considerations

Lead times: 2-12 months depending on scale and SKU.
Power readiness: confirmed before order (especially for B200 1000W+).
Networking: integrated procurement (switches, cables, NICs together).
Software licenses: NVIDIA AI Enterprise, vGPU licenses, CUDA support.
Service contracts: NVIDIA Mission Critical, 24/7 support for production.

The total cost of an AI deployment is more than just the GPU sticker price.

Real-world deployments and case studies

Examples of how organizations are using these GPUs.

Case 1: SaaS startup (small scale)

Setup: 16 H100s on CoreWeave, 1-year reserved.
Workload: Llama-3 70B inference for 100k MAU SaaS app.
Cost: ~$45k/month.
Stack: vLLM with FP8.

Case 2: Mid-market enterprise (medium scale)

Setup: 64 H100s on AWS, mix of on-demand and reserved.
Workload: multi-tenant inference + light fine-tuning.
Cost: ~$200k/month.
Stack: SGLang for chat workloads, vLLM for batch.

Case 3: AI-first company (large scale)

Setup: 512 H100s + 64 H200s on dedicated CoreWeave.
Workload: 100M tokens/day inference + multi-tenant fine-tuning.
Cost: ~$1.5M/month.
Stack: TRT-LLM for max performance, multi-region.

Case 4: Frontier lab (frontier scale)

Setup: 16,000 H100s + 1,000 B200s in dedicated datacenter.
Workload: pre-training, post-training, internal serving.
Cost: ~$50M/month total infrastructure.
Stack: Megatron-LM for training, custom inference for serving.

Case 5: Research institution

Setup: 32 H100s on academic cloud allocation.
Workload: research experiments, no production serving.
Cost: $20k/month (heavily subsidized).
Stack: PyTorch FSDP, Lightning.

Lessons across cases

Cost scales superlinearly with scale. Frontier-scale ops engineering is expensive.
Hardware mix matters. Most production uses 80-90% H100 even in 2026.
Cloud vs on-prem economics flip around 1000+ GPUs.
Multi-region adds 30-50% cost but reduces latency and improves availability.

Architecture comparison: NVIDIA vs alternatives

Detailed comparison of NVIDIA datacenter GPUs vs major alternatives.

NVIDIA H100/H200/B200 (2026 standard)

Strengths:

Mature CUDA ecosystem.
All major frameworks optimized.
Comprehensive software support.
Reference cluster designs.

Weaknesses:

Premium pricing.
Supply tightness for Blackwell.
Lock-in to CUDA.

Best for: anything mainstream.

AMD MI300X / MI355X

Strengths:

30-50% cheaper than H100 at similar performance.
192 GB HBM (more than H100, similar to B100).
ROCm ecosystem improving rapidly.

Weaknesses:

Smaller community.
Tooling lags NVIDIA.
Some optimizations not yet ported.

Best for: cost-sensitive inference, organizations with ROCm expertise.

Google TPU v5p / v6

Strengths:

Excellent performance for transformer training.
Tight JAX integration.
Available on GCP.

Weaknesses:

Locked to Google Cloud.
Not for PyTorch (or limited).
Different programming model.

Best for: GCP customers using JAX, training Google's own models.

AWS Trainium / Inferentia

Strengths:

Cheaper than H100 on AWS.
Tight AWS integration.
Production-ready.

Weaknesses:

Limited to AWS.
Smaller community.
Software ecosystem less mature.

Best for: AWS-native deployments where cost is critical.

Apple Silicon (M-series)

Strengths:

Excellent for local development.
Unified memory simplifies many things.
Energy efficient.

Weaknesses:

Not for production multi-tenant serving.
Single-node.

Best for: local development, prototyping, edge deployment.

Cerebras CS-3

Strengths:

Wafer-scale chip.
Unique architecture for some workloads.
Strong for batched inference.

Weaknesses:

Niche.
Limited production deployments.
Different programming model.

Best for: specific research and specialized workloads.

Groq

Strengths:

Ultra-low-latency inference.
LPU architecture optimized for token-by-token serving.

Weaknesses:

Inference-only.
Niche.

Best for: latency-critical inference workloads.

Pick by workload

General-purpose AI: NVIDIA.
Cost-sensitive inference: NVIDIA or AMD.
AWS-native: NVIDIA on EC2 or Trainium for cost.
Google Cloud + JAX: TPU.
Local dev: Apple Silicon.
Frontier training: NVIDIA (everyone does).
Latency-critical: Groq for some workloads.

For most teams, NVIDIA remains the safest choice through 2026-2027.

Hardware-software co-design

A subtle but important point: GPU performance depends on the software stack as much as the hardware.

CUDA evolution

CUDA's APIs evolve with hardware. Each new generation adds features:

Hopper: TMA, Async Memory Operations, Cluster Launches.
Blackwell: Memory Distribution Service, FP4 instructions.
Rubin: speculation primitives, more async.

Software has to use these to get full performance. PyTorch and frameworks are continuously updated.

Library stack

A complete AI training/serving stack:

CUDA: GPU programming primitive.
cuBLAS: matrix operations.
cuDNN: deep learning operations.
NCCL: collective communication.
TensorRT: inference optimization.
Transformer Engine: FP8 training.
FlashAttention: memory-efficient attention.
vLLM/SGLang/TRT-LLM: inference engines.

Each layer is updated as new hardware ships. Keeping the stack current matters.

Forward compatibility

CUDA generally maintains forward compatibility: code compiled for older GPUs runs on newer ones. The reverse isn't true (newer code may use features old GPUs don't have).

For deploying on mixed clusters: target the lowest-common-denominator GPU.

Backward compatibility

Hardware features sometimes get deprecated. Older GPUs may lose support in current CUDA versions.

For very old GPUs (P100, V100): use older CUDA toolkit versions.

Co-design example: FP8 + Transformer Engine

H100 added FP8 tensor cores. Transformer Engine library co-designed:

Hardware: FP8 GEMMs at 2× FP16 throughput.
Software: per-tensor scaling, automatic calibration, layer-specific routing.

Without TE, FP8 hardware is hard to use. With TE, it's a flag flip.

This co-design pattern is how NVIDIA stays ahead: hardware enables capability, software makes it accessible.

Open vs proprietary

CUDA is proprietary to NVIDIA. AMD's ROCm is open-source.

For teams concerned about lock-in: factor this into multi-vendor strategies. For most: CUDA's maturity outweighs lock-in concerns.

Programming model and CUDA evolution

Understanding the programming model that lets you actually use these GPUs.

CUDA basics

CUDA is the programming model for NVIDIA GPUs. Layers:

Kernel: function that runs on the GPU, executed by many threads in parallel.
Grid/Block/Thread: hierarchical organization. Threads in the same block share memory; blocks in the same grid don't.
Streams: queues of operations. Different streams can execute in parallel.

For most users, you don't write CUDA directly. PyTorch / JAX / TensorFlow handle it for you. But understanding helps debugging.

What changed across generations

Hopper (H100):

Thread Block Cluster: groups multiple blocks for shared memory access.
TMA (Tensor Memory Accelerator): asynchronous memory operations.
Distributed Shared Memory: cluster-level shared memory.

Blackwell (B200):

Native FP4 instructions.
Memory Distribution Service: more efficient cross-block memory.
Cooperative Group launches: better TP synchronization.

These features aren't always exposed via PyTorch APIs. Custom kernels (FlashAttention, Liger Kernel) leverage them for performance.

When to write custom kernels

For most teams: never. Use PyTorch's optimized ops, FlashAttention, etc.

For specific use cases:

Novel attention patterns.
Custom quantization formats.
Workloads where standard ops underperform by >20%.

NVIDIA's CUTLASS and Triton make custom kernels more accessible than raw CUDA.

Triton: the modern alternative

Triton is a Python-like DSL for writing GPU kernels. Less performant than handcrafted CUDA but much easier to write.

@triton.jit
def matmul_kernel(a_ptr, b_ptr, c_ptr, M, N, K, ...):
    # Triton kernel for matmul
    pid = tl.program_id(axis=0)
    ...

For teams that need custom kernels but don't have CUDA expertise, Triton is the answer.

CUDA driver vs runtime

CUDA Driver API: low-level, more control. CUDA Runtime API: higher-level, easier.

Most user code uses Runtime. Driver only needed for unusual scenarios.

Forward compatibility

CUDA generally maintains forward compatibility: code compiled for older architectures runs on newer ones. The reverse isn't true.

For deploying on mixed clusters: target the lowest-common-denominator GPU.

Practical maintenance and operations

Keeping GPU clusters healthy.

Health checks

Daily: nvidia-smi per GPU. Watch for:

ECC errors (rare but indicates hardware issue).
Temperature anomalies (>85C sustained).
Clock throttling.

Weekly: NCCL benchmark to verify cluster fabric health.

Monthly: thorough review of error counters, drift in performance metrics.

Common operational issues

ECC errors increasing: HBM degradation. Plan for replacement.

Temperature alerts: cooling failure or airflow issue. Investigate.

Clock throttling: power limits or thermal limits. May indicate facility issue.

PCIe errors: cable or slot issue. Reseat, replace if persistent.

NVLink errors: rare but possible. May require board replacement.

Driver and firmware management

Update strategy:

Stable: stick with one version per cluster generation.
Test new versions in staging before production.
Plan upgrades during maintenance windows.

Don't auto-update. Each version may have regressions.

Decommissioning

GPUs eventually retire. Common reasons:

Newer generation is more cost-effective.
ECC errors indicate end of life.
Thermal degradation.

For datacenter GPUs: useful life 4-7 years. Most retired after 3-5 due to economic obsolescence.

Sale/transfer of GPUs

Used GPU market is real. Datacenter operators sell off retired hardware.

Buyer beware: ECC error history isn't always disclosed. NVIDIA support doesn't transfer for some products.

For most: buy new. Used GPUs only if cost is critical and you can verify hardware health.

Software stack tooling

The libraries and frameworks you'll interact with.

Foundation libraries

CUDA Toolkit: compiler, libraries, drivers.
cuDNN: deep learning primitives.
NCCL: collective communication.
TensorRT: inference optimization.
cuBLAS: linear algebra.

These are NVIDIA's foundational stack. All AI software builds on them.

Frameworks

PyTorch: dominant deep learning framework.
JAX: research-oriented, strong on TPU.
TensorFlow: legacy enterprise deployments, declining.

For new projects in 2026: PyTorch is the safe default.

Inference engines

vLLM: most popular open-weight inference.
SGLang: structured workloads.
TensorRT-LLM: max performance on NVIDIA.
TGI: HF integration.
LMDeploy: Chinese open-weight specialist.
llama.cpp: local/CPU/edge.

See LLM Serving guide for selection.

Training frameworks

Megatron-LM: NVIDIA's reference for TP/PP.
DeepSpeed: ZeRO-style sharding.
NeMo: NVIDIA's high-level wrapper.
PyTorch FSDP: native sharding.
Lightning: research-oriented.

See Distributed Training guide.

Specialized libraries

FlashAttention: memory-efficient attention.
Transformer Engine: FP8 training/inference.
Triton: GPU kernel DSL.
CUTLASS: optimized matrix multiplication.
Liger Kernel: fused LLM kernels.

Most users encounter these through frameworks; rarely directly.

Observability

NVIDIA DCGM: data center GPU manager.
NVIDIA Nsight Systems: profiling.
Prometheus + Grafana: metrics.
Weights & Biases: training metrics.

For production deployments: monitor everything.

Vendor-managed services

AWS Bedrock: managed inference on AWS hardware.
Azure AI: similar on Azure.
GCP Vertex AI: similar on GCP.
NVIDIA NIM: containerized inference.

For teams without infrastructure expertise: managed services trade cost for simplicity.

How Hopper changed AI training economics

The H100 was not just a faster GPU — it changed what was economically feasible.

Before H100: A100 era (2020-2022)

GPT-3 (175B) was trained on roughly 10,000 V100s. The training run took months and cost an estimated $4-12M. Most labs couldn't afford this.

A100 in 2020 made things ~3× cheaper but still expensive. A 70B model required ~1,000 A100s for reasonable training time.

After H100: 2022-2025 era

H100 with FP8 made transformer training ~3× faster than A100. A 70B model could be trained on 256 H100s in a week or two.

This put frontier-scale training within reach of more organizations:

Mid-size labs (Mistral, AI21, Cohere).
Tech companies (Adept, Inflection, etc.).
Academic consortia.

The result: explosion of capable open-weight models. Llama, Mistral, Qwen, etc. were enabled by H100 economics.

Blackwell era: 2024+

B200 brings another 2-3× improvement. A 70B model now trains on ~100 B200s in a week.

This continues democratizing frontier training. By 2027, training a competitive 70B model from scratch may be a $1-2M effort, accessible to many organizations.

The implications

GPU economics drive AI capability. As compute gets cheaper:

More organizations can train models.
Open-weight models become more competitive.
Specialization increases (domain-specific models).
Frontier moves higher (now 1T+ parameters).

The industry's pace is dictated more by compute access than by algorithm research.

Buying advice by use case

Distilled recommendations.

Solo developer / researcher

Local: Apple Silicon Mac (M-series) for development. Rent cloud GPUs as needed.
Cloud: Lambda on-demand H100 or RunPod. $1-3/hour.
Don't buy hardware unless you have stable usage 6+ months.

Startup at small scale

Use cloud (AWS, GCP, Lambda).
1-year reserved if usage is predictable.
Test with multiple providers; performance varies.

Mid-size company

Mix of cloud reserved and on-demand.
Consider CoreWeave or Lambda for sustained workloads.
Evaluate on-prem if usage exceeds 24/7 for 2+ years.

Enterprise

Long-term cloud reservations or dedicated cloud.
On-prem deployment with NVIDIA-blessed reference architectures.
Multi-region for resilience.
Engineering team to operate.

Research institution

Academic cloud allocations (subsidized).
Shared cluster facilities (university, NSF).
DGX SuperPOD for serious work.

AI lab/foundation lab

Direct NVIDIA partnership.
Custom datacenter or co-location.
Multi-year capital investment.
Significant operations team.

For each level, the answer scales with usage. Start small and scale up.

Common buying mistakes

Things teams get wrong when procuring GPUs.

Buying for peak instead of sustained

Sizing GPU capacity for the rare peak load. Wasting compute most of the time.

Right approach: size for sustained load. Use cloud burst for peaks.

Underestimating networking

Buying lots of GPUs without commensurate networking. Cluster underperforms.

Right approach: spec networking proportional to compute.

Ignoring power and cooling

Hardware budget without facility upgrade. Can't actually deploy what you bought.

Right approach: verify facility readiness before procurement.

Single-vendor lock-in without verification

Going all-NVIDIA without testing alternatives that might fit some workloads.

Right approach: at least benchmark alternatives. Even if you go NVIDIA, having data is good.

Buying current-gen when next-gen is imminent

Purchasing H100 when B200 is 6 months away. New gen makes purchase look bad.

Right approach: factor in roadmap. Sometimes wait, sometimes buy now.

Over-provisioning HBM

Buying H200 when H100 80GB is sufficient. Paying premium for unused capacity.

Right approach: profile actual KV cache usage. Buy what you need.

Under-provisioning HBM

Buying H100 80GB and discovering you need 141GB or 192GB for long-context.

Right approach: profile representative workload before procurement.

Skipping operational readiness

Procuring hardware without planning monitoring, alerts, on-call.

Right approach: operational plan in parallel with procurement.

Hardware-aware model design

How GPU architecture influences model design.

GQA vs MQA vs MHA

GQA (Grouped-Query Attention) reduces KV memory. Makes long-context serving viable on H100/H200. Most modern open-weight models use GQA-8.

Without GQA, Llama-3 70B-class long-context serving wouldn't fit affordable hardware. Architecture and hardware co-evolve.

MoE economics

MoE models have many parameters but only some are active per token. Training compute is similar to dense models with same active parameters; serving is more memory-intensive.

GPU choice affects MoE economics: H200/B200's larger HBM is well-suited.

FP8 native architectures

Some 2024-2026 models are designed with FP8 in mind. Architectural choices that work better with FP8:

Per-channel quantization-friendly weight distributions.
Numerically stable activations.
Layer norm rather than batch norm.

This is becoming standard for new models.

Attention head sizing

head_dim = 128 is universal in 2026. This wasn't always — earlier models used 64.

Why 128: matches Hopper's tensor core dimension preferences. Smaller dimensions waste tensor core capacity.

Architecture choices reflect hardware optimization.

Sliding window attention

Mistral and others use sliding-window attention to bound KV cache. Useful when models have to fit on limited HBM.

Architectural choice driven by hardware constraint.

Hybrid architectures

Mamba/transformer hybrids (Jamba) work well on H200/B200's large HBM. Specific layer ratios optimized for current GPUs.

Hardware progression enables more diverse architecture experimentation.

Conclusion: how to think about GPU choice

Picking the right GPU isn't about specs. It's about workload fit, economics, and operational considerations.

The decision tree:

What's the workload? Training, inference, mixed.
What scale? Single GPU, single node, multi-node, frontier.
What's the latency budget?
What's the cost target?
What's the operational complexity tolerance?
What's the hardware availability for your timeline?

Most teams in 2026:

Inference: H100 for cost, H200 for long-context.
Training: H100 for fine-tuning, B200 for new pre-training.
Frontier: B200/GB200 if you can get it, H100 if not.

The most valuable advice: benchmark your actual workload. Marketing TFLOPs are upper bounds. Real workload throughput is the only metric that matters.

NVIDIA's lead in 2026 is real but not absolute. AMD MI300X is competitive for inference. Specialized chips (Cerebras, Groq) win in niches.

For most: NVIDIA is the safe choice. The ecosystem maturity outweighs marginal cost or performance differences.

Evaluate. Test. Benchmark. Decide.

Frequently misunderstood specs

Things in spec sheets that often confuse buyers.

Peak vs sustained TFLOPs

Marketing numbers (1979 TFLOPs FP16 on H100) are theoretical peak with optimal conditions.

Real workload sustained: 40-60% of peak.

For LLM training: ~700 TFLOPs sustained on H100 (vs 1979 peak). Plan with sustained.

HBM bandwidth quoted vs achieved

Quoted: H100 has 3.0 TB/s HBM bandwidth.

Achieved: 2.7-2.9 TB/s sustained on memory-bound workloads.

Difference comes from access patterns and overhead. Plan with 90% of quoted.

NVLink directional

NVLink bandwidth is per-direction. 900 GB/s on H100 means 900 GB/s send + 900 GB/s receive simultaneously.

For collective operations, often quoted as bidirectional (1.8 TB/s for H100). Be careful comparing numbers.

TDP vs typical power

TDP = thermal design power = maximum sustained power.

Typical AI workload: 70-90% of TDP. So H100 700W TDP draws 500-630W typical.

For datacenter sizing: use typical for capacity planning, TDP for safety margin.

Compute precision differences

FP16, BF16 = 2 bytes. Same throughput on H100/B200.

But FP8 (1 byte) has 2× the FP16 throughput. FP4 has 4×. The per-byte speedup is what matters.

Capacity vs usable capacity

H100 has 80 GB HBM but ~75 GB is usable after CUDA reserved memory. Plan around usable.

Same for all GPUs: subtract 5-8 GB from quoted for actual workload memory.

NVL variant capacity

H100 NVL has 94 GB total but it's split between two physical GPUs (47 GB each visible to software). Different from a single 94 GB GPU.

Affects how the workload sees memory.

These quirks matter for capacity planning. Read spec sheets carefully.

Tracking GPU evolution: signals to watch

How to stay ahead of GPU generations.

Public signals

NVIDIA earnings calls: announcement of next-gen products.
GTC keynotes: detailed architectural disclosures.
Hyperscaler quarterly reports: GPU procurement signals.
TSMC capacity reports: indicates supply for upcoming generations.

Industry events to follow

GTC (annual): NVIDIA's flagship conference.
ISC HPC (annual): supercomputing trends including AI.
HotChips (annual): chip architecture deep dives.
AI Hardware Summit: industry-wide.

Practical lead times

New generation announcement: ~6 months from broad availability.
Reference architecture: ~12 months.
Stable production deployments: ~18 months.

Plan procurement with 12-18 month horizon for current-gen, longer for next-gen.

Software readiness signals

When new GPU is ready for production:

PyTorch updates with full support.
vLLM/SGLang/TRT-LLM versions tested.
FlashAttention version supports the hardware.
NCCL optimized for the new fabric.

If these are still rough, hardware isn't quite ready for production.

Pricing trends

Watch for pricing pressure from:

Newer generation launches (older drops in price).
AMD/Cerebras/etc. competition.
Cloud provider undercutting.

For long-term planning: assume 30-50% price reduction over 18 months for any given generation.

This is a fast-moving market. Monthly attention pays off.

Hardware vs algorithm tradeoffs

Sometimes algorithm improvements (FlashAttention, mixed precision, sparse attention) deliver more value than hardware upgrades. Don't always wait for the next gen.

For a given workload: profile current hardware, identify bottlenecks, choose hardware vs algorithm investment based on actual data.

The best teams in 2026 invest in both: hardware refresh on schedule, plus algorithm research that keeps pace.

Watch for new entrants

Beyond NVIDIA-AMD-Intel, new entrants emerge periodically:

Specialized AI accelerator startups.
Hyperscaler in-house silicon (AWS Trainium, Google TPU, Microsoft Maia).
Geopolitical alternatives (China-domestic chips).

Most don't displace incumbents. A few find sustainable niches.

For most teams: track but don't bet on new entrants without strong evidence they'll work for your workload.

Geographic considerations

Different regions have different GPU access:

US/EU: full NVIDIA availability.
China: export restrictions; H100 Chinese variants and domestic alternatives.
Other regions: typical secondary cloud provider availability.

For international deployments: factor in regional restrictions and supply.

How to evaluate new GPU offerings

When a new SKU launches:

Wait 3-6 months for ecosystem maturity.
Run your workload benchmarks.
Compare cost-per-token (or cost-per-training-step).
Verify availability and pricing stability.
Pilot deployment with small portion of traffic.
Ramp up if successful.

Don't be first to deploy unless you have specific reasons. The early-adopter premium rarely pays off.

The competitive landscape

NVIDIA's dominance in 2026 is real but not absolute.

NVIDIA's strengths

Software ecosystem (CUDA, cuDNN, NCCL, Megatron, NeMo, TRT-LLM).
Reference architectures (DGX, HGX, GB200 NVL72).
Datacenter relationships and supply.
90%+ of frontier training runs.

Where competitors compete

AMD MI300/MI355: real alternative for inference. Software is catching up (vLLM, SGLang, PyTorch ROCm all work). Pricing is 30-50% cheaper. Some cloud providers offer it.

Google TPU: dominant within Google's ecosystem (JAX, TensorFlow). Not directly available outside GCP. Excellent for training when workload fits.

Apple Silicon (M-series): dominant for local inference. Not competitive for production multi-tenant serving but excellent for development.

Cerebras: niche but interesting. Wafer-scale chip with deterministic compute. Some training workloads benefit; mainstream is still NVIDIA.

Groq: specialized inference accelerator. Ultra-low latency for some workloads. Not competitive for training.

Custom silicon (Tesla Dojo, AWS Trainium, Amazon Inferentia, Meta MTIA): tied to specific platforms. Not user-accessible in the way NVIDIA is.

What could shift NVIDIA's dominance

Software ecosystem catches up on AMD: removes one of NVIDIA's biggest advantages.
Open-weight model ecosystem standardizes on a stack that's hardware-agnostic (e.g., MLX-style abstractions): reduces lock-in.
Power/heat constraints favor specialized chips: NVIDIA's compute density may not scale forever.
Anti-trust action: regulatory pressure might affect bundling.

In 2026, NVIDIA's lead looks safe through 2028 at least. Competitors are improving but not closing the gap fast enough.

Multi-vendor strategies

Some sophisticated buyers are diversifying:

Frontier labs run multi-vendor (NVIDIA primary + AMD or TPU secondary) to hedge.
Cost-sensitive deployments use AMD where ecosystem is mature enough.
Edge deployments use whatever fits the form factor and power envelope.

For most teams, full NVIDIA is still simplest. Diversification has operational cost; it's worth it only at scale.

Q: How do I evaluate a new GPU SKU?

Real-world workload throughput is the only metric. Run vLLM benchmark on your actual workload. Marketing TFLOPs are upper bounds, not predictions.

Glossary

Ampere: 2020 GPU architecture (A100).
Blackwell: 2024 GPU architecture (B100, B200, GB200).
DGX: NVIDIA's reference 8-GPU server.
FP4 / FP8 / FP16 / BF16: floating-point formats.
GB200 NVL72: rack-scale Blackwell with 72 GPUs in one NVLink fabric.
HBM: high-bandwidth memory.
Hopper: 2022 GPU architecture (H100, H200).
NVL: NVLink-connected, dual-board (e.g., H100 NVL).
NVLink: NVIDIA's GPU interconnect.
NVSwitch: NVLink fabric switch enabling all-to-all GPU communication.
PCIe: Peripheral Component Interconnect Express. Slower than NVLink.
Rubin: announced 2026 GPU architecture (R100, GB300).
SXM: NVLink-connected GPU socket form factor (DGX-style).
Tensor core: matrix-multiply unit on the GPU.

References

Foundational research

Mixed-precision training — Micikevicius et al., 2017. "Mixed Precision Training." arXiv:1710.03740. The original FP16 + loss-scaling recipe that all modern tensor-core training inherits from.
FP8 formats for deep learning — Micikevicius et al., 2022. "FP8 Formats for Deep Learning." arXiv:2209.05433. Defines E4M3 / E5M2 — the numerics implemented by Hopper's Transformer Engine.
FlashAttention-2 — Dao, 2023. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." arXiv:2307.08691. The kernel that made long-context training tractable on Ampere and Hopper.
FlashAttention-3 — Shah et al., 2024. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision." arXiv:2407.08608. Uses Hopper TMA + warp specialization to hit ~75% of peak FP16.
NVIDIA A100 Tensor Core GPU — Choquette et al., IEEE Micro 2021. IEEE Xplore. The reference architecture description of the generation Hopper succeeded.

Production systems and vendor documentation

NVIDIA, H100 Tensor Core GPU Architecture Whitepaper, 2022. Resource page.
NVIDIA, Hopper Architecture Product Page, 2022. nvidia.com/h100.
NVIDIA, Blackwell Architecture Technical Brief, 2024. nvidia.com/blackwell.
NVIDIA, H200 Tensor Core GPU Datasheet, 2023.
NVIDIA, GB200 NVL72 Reference Architecture, 2024.
NVIDIA, Rubin GPU Roadmap Update, GTC 2024 keynote.

Background reading

Jouppi et al., 2017. "In-Datacenter Performance Analysis of a Tensor Processing Unit." arXiv:1704.04760. The TPU paper that catalyzed datacenter-scale AI accelerator design.
Patterson et al., 2021. "Carbon Emissions and Large Neural Network Training." arXiv:2104.10350. Frames the power-per-FLOP context that drives every generation transition.
Hooker, 2020. "The Hardware Lottery." arXiv:2009.06489. Why transformer-shaped hardware (FP8, TMA, NVLink) has shaped which models exist.

Arithmetic intensity and the roofline model

The single most useful mental model for predicting GPU performance on a workload is the roofline: plot peak FLOPs on one axis, peak HBM bandwidth on the other, and an operation's arithmetic intensity (FLOPs per byte read from HBM) tells you which one bounds it.

Per-GPU roofline numbers

GPU	Peak FP8 TFLOPs	HBM BW (TB/s)	Roofline knee (FLOPs/byte)
A100	624 (FP16)	1.5	416
H100	3958	3.0	1319
H200	3958	4.8	824
B100	7000	7.7	909
B200	9000	8.0	1125

The "knee" is the arithmetic intensity above which a workload is compute-bound and below which it's bandwidth-bound. H100's FP8 knee at 1319 FLOPs/byte means an operation needs 1319 FP8 multiply-accumulates per byte of HBM read to saturate the tensor cores. Below that, you're starving the cores.

Where common ops land

Dense GEMM (training matmuls): 100-2000 FLOPs/byte depending on M/N/K. Large training GEMMs are compute-bound; small ones (LM head, output projection at batch=1) are bandwidth-bound.
Attention (FlashAttention): ~100-200 FLOPs/byte. Bandwidth-bound on Hopper and Blackwell. Why FlashAttention-3 focuses on async copies and warp specialization rather than reducing FLOPs.
LM decode (BS=1): ~2 FLOPs/byte. Catastrophically bandwidth-bound. This is why decode tokens/sec scales linearly with HBM bandwidth and not with FLOPs.
Decode at BS=64: ~80-100 FLOPs/byte. Still bandwidth-bound on H100/B200; this is the regime continuous batching exploits.
Optimizer step (AdamW): ~1-2 FLOPs/byte. Pure bandwidth.

What the roofline says about GPU choice

If your workload's arithmetic intensity is well below the knee, doubling FLOPs doesn't help — only HBM bandwidth does. This is why H200 (same compute as H100, 1.6× bandwidth) is a 1.5× speedup for inference but a wash for training. It's also why B200's combined FLOPs and bandwidth bump matters: both axes moved.

Per-workload performance ceilings

Calibrated expectations for what each GPU should hit on real workloads, with sustained (not peak) numbers.

Dense 70B inference, BF16, single request

H100 SXM: 28-32 tok/s decode, 200-300 ms TTFT (4k prompt).
H200 SXM: 42-48 tok/s decode (1.5× from HBM), similar TTFT.
B200 SXM: 55-65 tok/s decode, 1.6× TTFT improvement.

Numbers from vLLM 0.6+, FP16 KV cache, 4k context. Decode is HBM-bandwidth-bound; the ratios match HBM bandwidth ratios within 10%.

Dense 70B inference, FP8 weights + FP8 KV, batch=32

H100: 1800-2200 aggregate tok/s.
H200: 2800-3300 aggregate tok/s (capacity headroom lets bigger batches).
B200: 4500-5500 aggregate tok/s (capacity + FP8 throughput).

Dense 70B fine-tuning (one step), 16 GPUs

H100 ×16: ~1.8 sec/step, 700 TFLOPs/GPU sustained (35% of 1979 FP16 peak).
B200 ×16: ~0.9 sec/step, 2200 TFLOPs/GPU sustained (40% of 4500 FP16 peak).

MoE 8×22B inference (Mixtral-style), TP=8

H100 ×8: ~2500 tok/s aggregate.
B200 ×8: ~6000 tok/s aggregate (capacity lets active experts stay resident).

Diffusion image generation, SDXL-class

H100: 18-22 images/sec at 1024×1024 with batch=8.
B200: 35-42 images/sec.

Diffusion has higher arithmetic intensity than LLM decode; the bump is closer to the FLOPs ratio than the bandwidth ratio.

Total cost of ownership math

Sticker price is a fraction of TCO. The full math by component, for a 1000-GPU on-prem H100 deployment versus equivalent cloud reservation over three years.

On-prem 1000× H100 (SXM, HGX nodes)

Hardware: 125 nodes × ~$300k = $37.5M.
Networking: 64 IB switches + 1000 NICs + optics ≈ $4M.
Power: 700W × 1000 GPUs + ~500W overhead per GPU × 24/7 × 3 years × $0.08/kWh ≈ $2.5M.
Cooling: 30% of power = $0.75M.
Facility: $1-3M depending on whether retrofitting.
Ops: 5 FTE × $250k × 3 years = $3.75M.
Three-year TCO: ~$50M.
Effective $/GPU-hour: $50M / (1000 × 24 × 365 × 3) ≈ $1.90.

Three-year reserved cloud H100 at $2.00/hr

1000 GPUs × $2/hr × 24 × 365 × 3 = $52.6M.
Zero capex, zero ops headcount, full elasticity.

When each wins

At sticker prices in 2026, three-year reserved cloud roughly matches well-run on-prem. On-prem wins decisively when you keep the hardware four+ years (depreciation continues; cloud doesn't get cheaper) and when you negotiate hardware below list. Cloud wins decisively when utilization is variable or growing — paying for unused on-prem capacity destroys the math.

B200 changes the calculus

B200 nodes cost ~$500k each. Three-year reserved cloud B200 at $4.50/hr is $118M for 1000 GPUs. On-prem B200 with similar facility overhead is ~$80M — but only if you can hit 90%+ utilization. The break-even shifts toward cloud at higher SKUs because cloud lets you absorb both capex risk and supply uncertainty.

Hardware-software gotchas by generation

Each new GPU generation introduces footguns that bite teams who treat it as a drop-in replacement.

H100 gotchas

FP8 quality regression at long context: per-tensor scaling overshoots on attention outputs past 16 K tokens. Fix: switch attention to BF16 or use per-channel scaling.
NVSwitch firmware bugs in early DGX H100: occasional 5-10% NVLink bandwidth drop on specific message sizes. Fixed in firmware 96.10.84+.
GPUDirect RDMA requires PCIe topology match: GPU and NIC must share a PCIe switch. Auto-cabled DGX-H100 is correct; custom HGX builds need verification.

H200 gotchas

Power profile differs from H100 despite same TDP: H200 spends more time at peak power (HBM is more aggressive). Cooling sized for H100 sometimes throttles H200.
NCCL needs 2.20+: older NCCL detects H200 as H100 and misses HBM bandwidth, hurting AllReduce performance.

B200 gotchas

1000W TDP requires liquid cooling for most racks: air-cooled deployments throttle quickly. Plan facility before procurement.
NVLink-5 mixed clusters degrade to NVLink-4 speed: don't mix B200 and H100 in the same TP group.
FP4 quality requires careful calibration: out-of-the-box FP4 quantization on instruction-tuned models drops MMLU by 2-4 points. AWQ or LLM-Compressor with B200-aware kernels recover most of it.
CUDA 12.4+ required: older CUDA toolkits don't expose B200 FP4 instructions.

GB200 NVL72 gotchas

Single point of failure on cooling distribution units: a CDU failure can take down 72 GPUs at once. Redundant CDU procurement is non-negotiable for production.
Rack-scale NVLink amplifies topology bugs: a misconfigured switch can silently halve effective bandwidth. nccl-tests at TP=72 is a mandatory acceptance gate.
Firmware updates are disruptive: NVLink switch firmware updates require the whole rack offline. Plan maintenance windows.

Comparing GPUs to alternatives in 2026

Detailed numbers for cross-vendor comparisons.

NVIDIA H100 vs AMD MI300X

Spec	H100 SXM	MI300X
HBM	80 GB HBM3	192 GB HBM3
HBM BW	3.0 TB/s	5.3 TB/s
FP16 TFLOPs	1979	1307
FP8 TFLOPs	3958	2614
Interconnect	NVLink-4 900 GB/s	xGMI 896 GB/s
TDP	700 W	750 W
Software	CUDA + cuDNN + NCCL	ROCm + MIOpen + RCCL
Price (mid-2026)	$25-30k	$15-20k

MI300X has more HBM and more bandwidth; H100 has more compute and the software ecosystem. For inference of models that fit in MI300X's 192 GB without TP, AMD often wins on cost-per-token. For training and complex multi-GPU pipelines, NVIDIA wins on operational maturity.

NVIDIA B200 vs AMD MI350X

MI350X (late 2025) is AMD's response to B200: 288 GB HBM3e, 8 TB/s, FP4 support. On paper competitive. Real-world: ROCm's FP4 implementations are 6-12 months behind NVIDIA's Transformer Engine. Production deployments at scale still favor B200 by mid-2026; MI350X adoption is real but slower.

NVIDIA H100 vs Google TPU v5p

TPU v5p offers similar peak BF16 FLOPs (459 TF/chip vs H100's 989 — but v5p chips are paired). Per-pod (256 chips), v5p delivers slightly better $/FLOP than equivalent H100 cluster on GCP. The cost is JAX-only programming and GCP lock-in.

NVIDIA H100 vs AWS Trainium2

Trainium2 (2024) targets training-only workloads. Per-chip FP8 TFLOPs ≈ 1300; per-trn2.48xlarge (16 chips) cost ≈ 30-40% below equivalent p5.48xlarge. Catch: AWS Neuron SDK is required and lags PyTorch's NVIDIA path by ~6 months on new model architectures.

When not to upgrade

Counter-intuitive cases where sticking with older hardware wins.

Inference workloads where bandwidth is not the bottleneck

If your model is small (7B-13B), fits in 24 GB HBM, and runs at low concurrency, an A10 or L40S delivers 80-90% of H100 throughput at 20-30% of the price. The bandwidth premium of H100 only pays off for memory-bound workloads.

Long-tail batch processing

For workloads where latency doesn't matter (overnight document processing, bulk embedding generation), older GPUs at spot prices deliver better $/token than current-gen on-demand. A100 spot at $0.40/hr for 8-hour batch jobs is hard to beat.

Workloads constrained by host CPU or storage

If your bottleneck is data loading or preprocessing, upgrading GPUs doesn't help. Buy more storage bandwidth or upgrade CPUs first. The benchmark to run is GPU utilization during your workload — if it's below 60%, the bottleneck is elsewhere.

Research with frequent code rewrites

When your code changes weekly, ecosystem maturity matters more than peak FLOPs. H100 has every kernel optimized; B200 has fewer. For research iteration speed, H100 is often the better tool through 2026.

H200 versus B200 for inference: the detailed comparison

The most common SKU decision in 2026, with numbers.

Memory and bandwidth

H200: 141 GB at 4.8 TB/s. Holds Llama 70B BF16 with ~70 GB for KV cache (28 K context at 32 concurrent users, GQA-8). B200: 192 GB at 8 TB/s. Holds Llama 70B BF16 with ~120 GB for KV cache (50 K context at 32 concurrent users) or 70B FP4 with ~150 GB for KV cache.

Compute

H200: 3958 FP8 TFLOPs (peak). Same as H100. B200: 9000 FP8 TFLOPs, 18000 FP4 TFLOPs. Roughly 2.3× FP8 and 4.5× FP4 over H200.

When H200 wins

Workload doesn't need FP4 and quality is BF16/FP8.
Cost-per-token matters and H200 reserved is ~40% cheaper than B200 reserved.
Air-cooled facility (H200 fits, B200 typically doesn't).
Software stack hasn't been validated for B200 yet (older vLLM, custom kernels).

When B200 wins

High-concurrency serving where compute matters as much as bandwidth.
FP4 quality is acceptable (most chat workloads).
Frontier-scale training reused for inference between training runs.
Multi-year horizon — B200 has more runway before Rubin replaces it.

A specific decision matrix

Workload	H100	H200	B200
7B inference, BS=1	Best $/tok	Overkill	Overkill
70B inference, BS=1, 4K ctx	Acceptable	Best $/tok	Best abs perf
70B inference, BS=32, 32K ctx	Capacity-limited	Best balance	Best abs perf
405B inference	TP=8 minimum	TP=4 sufficient	TP=4 sufficient
7B fine-tuning	Best $/run	Wash	Overkill
70B fine-tuning	Acceptable	Acceptable	Best wall-clock
Frontier pre-training	Past its peak	Stopgap	Best choice
MoE inference (Mixtral 8×22B)	TP=8, cramped	TP=4, comfortable	TP=4, comfortable

Extended FAQ

Q: How does FP4 quality actually compare to FP8 on production benchmarks?

With AWQ or LLM-Compressor calibration on Llama 3.3 70B Instruct, FP4 loses 0.5-1.2 points on MMLU, 0.3-0.8 points on HumanEval, and 1-2 points on GSM8K versus FP8. Code generation is the most sensitive workload to FP4; chat and summarization the least. For most user-facing chat applications, FP4 is indistinguishable from FP8 in blind comparisons.

Q: When does it make sense to mix H100s and B200s in one cluster?

When you have separable workloads. Run inference on H100 (cheap, capacity-rich) and training on B200 (fastest available). Don't put them in the same TP or DP group — performance dictated by the slower GPU and topology becomes asymmetric. Schedulers like Kubernetes with node selectors keep them isolated cleanly.

Q: Is the GB200 NVL72 worth it over 9 separate B200 nodes?

For TP > 8 workloads, yes — rack-scale NVLink eliminates the IB bottleneck that kills cross-node TP. For TP ≤ 8 workloads, no — you get the same compute at lower operational complexity in 9 separate nodes. The break-even is whether you're training a 400B+ dense model or a large MoE.

Q: How much HBM headroom do I need for KV cache in serving?

Rule of thumb: weights consume params × bytes_per_param, leaving HBM_capacity - weights - 5 GB CUDA overhead for KV cache. Per-token KV cache bytes ≈ 2 × num_layers × num_kv_heads × head_dim × dtype_bytes. For Llama 70B GQA-8 with BF16 KV, that's ~328 KB per token per request. A 32 K context request consumes ~10.5 GB of KV cache. On H100 80 GB with 140 GB weights via TP=2, you have ~5-10 GB per GPU for KV cache before contention — supports ~5-10 concurrent 32 K requests. See KV cache inference memory math for the full calculation.

Q: What's the realistic supply situation for B200 in mid-2026?

Hyperscaler-allocated B200 is mostly committed through 2026; specialty GPU clouds (CoreWeave, Lambda) have 2-4 month lead times for new contracts; on-demand availability fluctuates daily. For non-strategic customers, expect 60-90 day allocation. Rubin sampling is starting Q4 2026, which will free B200 supply somewhat in 2027.

Q: Should I worry about PCIe bandwidth for inference?

Only if you're using PCIe H100 (not SXM) or running multiple inference workers per GPU that contend for host I/O. PCIe 5.0 x16 delivers 64 GB/s, which is fine for serving but bottlenecks training data loading on very fast storage. For all SXM/HGX configurations, PCIe is rarely the bottleneck — HBM and NVLink dominate.

Q: How does NVLink-C2C affect Grace+Hopper or Grace+Blackwell?

NVLink-C2C is the chip-to-chip interconnect between Grace CPU and Hopper/Blackwell GPU. 900 GB/s bidirectional. Practical effect: CPU memory accesses from GPU are 5-10× faster than over PCIe. Enables tractable CPU-side optimizer states for very large models (ZeRO-Infinity-style offload runs at NVLink speed). The cost: Grace+Hopper / Grace+Blackwell systems are 30-40% more expensive than x86+Hopper / x86+Blackwell at the same compute.

Q: What's the difference between DGX and DGX SuperPOD?

DGX is a single 8-GPU node. DGX SuperPOD is a reference architecture for 32-2048+ DGX nodes — pre-validated InfiniBand topology, storage, management software, and NVIDIA support. SuperPOD pricing is per-node + a SuperPOD entitlement; total premium is 10-20% over raw DGX procurement, in exchange for "it works on day one" guarantees and 24/7 NVIDIA Mission Critical support.

Q: Are there real workloads where AMD MI300X beats H100?

Yes. (1) Single-GPU inference of 100B+ dense models that fit in 192 GB MI300X but require TP=2 on H100 — MI300X wins on simplicity and cost. (2) Bandwidth-bound decode at small batch sizes — MI300X's 5.3 TB/s beats H100's 3 TB/s. (3) Workloads where ROCm has caught up with CUDA (vLLM inference of standard model architectures). MI300X is real for production inference of standard models; less so for novel architectures or training.

Q: How do I evaluate Hopper vs Blackwell for my workload without buying both?

Cloud spot pricing. Run identical vLLM benchmark on AWS p5.48xlarge (H100) and p6e (B200) for 4-8 hours at spot rates. Compare $/token and tokens/sec at your concurrency target. Cost: <$500. Most teams skip this and over- or under-buy by tens of thousands.

Q: What's the right GPU for fine-tuning a 70B Llama base?

For LoRA fine-tuning: 4-8× H100 80GB is sufficient — LoRA adds <1 GB per GPU and step time is dominated by forward/backward, not memory. For full fine-tuning: 16-32× H100 with ZeRO-3 or FSDP, or 8-16× B200 with the same. The cost difference is 30-50% in B200's favor when wall-clock matters; cloud time costs less than engineering iteration costs.

Q: Will Rubin (R100) make my Blackwell investment obsolete?

Not for inference — production inference clusters typically run for 3-5 years before refresh. Rubin will be 2× faster but B200 will still be economically viable through 2028. For training, Rubin will shift new pre-training runs but B200 will remain valuable for fine-tuning, eval, and inference well into 2028. Plan refresh on amortization schedule, not on hype cycle.

Q: Does NVIDIA NIM run on AMD or Intel?

No. NIM is NVIDIA-CUDA-only. The ROCm equivalent is loose — vLLM runs on AMD natively, but the packaging and runtime guarantees NIM provides aren't replicated on AMD or Intel. For NVIDIA shops it simplifies deployment; for multi-vendor shops it's not portable.

Q: How do I plan power for a 1000-GPU H100 cluster?

700W TDP × 1000 GPUs + 30% overhead (CPU, NIC, storage, fans) + 30% PUE for cooling = ~1.2 MW continuous IT load plus 360 KW cooling. That's a 1.5 MW facility commitment. Annual power cost at $0.08/kWh: ~$1M. For B200 1000-GPU: ~2.0 MW with PUE, $1.4M/year. These numbers shape facility selection more than they shape GPU selection.

Q: When should I use H100 NVL versus H100 SXM?

H100 NVL is a dual-board PCIe variant — 2× H100 dies sharing 94 GB HBM and NVLink between them, but no NVLink to other H100s in the host. Use it for 2-GPU inference workloads in standard PCIe servers where you can't justify SXM/HGX. For anything 4+ GPUs, SXM/HGX is cheaper per FLOP and more flexible.

Q: Is liquid cooling worth retrofitting for an existing H100 cluster?

Rarely. H100 is comfortable in air-cooled 8-9 kW racks. Liquid cooling pays off when you need to densify (>15 kW per rack) or when you're moving to B200/GB200 anyway. For pure H100, the retrofit cost ($50-100k per rack) doesn't pay back in efficiency gains.

Q: How does FP8 KV cache affect serving quality?

FP8 KV cache (E4M3 format) loses ~0.1-0.3 points on quality benchmarks versus BF16 KV when applied to Llama 3.x-class models. The win is 2× KV capacity at the same HBM, enabling 2× concurrent users or 2× context length. Almost always worth it for chat workloads; sometimes not for retrieval-style precision-critical applications.

Q: What's the right NVLink topology for an 8-GPU server?

NVSwitch all-to-all (DGX/HGX H100/B200 standard) — every GPU has the same 900 GB/s (H100) or 1.8 TB/s (B200) to every other GPU. Older topologies (hypercube, partial mesh) are deprecated. If a vendor offers anything other than NVSwitch all-to-all in 2026, they're shipping last-gen hardware.

Q: Should I worry about NVLink fabric reliability on GB200 NVL72?

Yes, modestly. The 72-GPU NVLink fabric is a single failure domain. NVIDIA's design includes fabric redundancy (multiple NVSwitch chips per rack, error correction), but a fabric event still degrades the whole rack. Production deployments include rack-level workload migration playbooks and N+1 rack capacity to absorb failures.

Q: How do I benchmark a new GPU SKU before committing?

Five-step procedure: (1) Run nccl-tests all_reduce_perf to verify fabric. (2) Run a vLLM offline-throughput benchmark on your actual model. (3) Run a continuous-batching latency test at your target concurrency. (4) Run a 24-hour soak to catch thermal issues. (5) Measure $/token end-to-end including idle time and warmup. Skip any step at your peril.

Q: What about GPU resale value?

H100 SXM modules retain ~50-60% of new price after 2 years on the secondary market. A100 retains ~30-40%. B200 is too new for resale data but expected to be similar to H100. Resale market is real but caveat-heavy — no NVIDIA support transfers, no guarantee against ECC degradation, no warranty.

Q: Why is my GPU utilization stuck at 60-70%?

Most common causes ranked: (1) Data loading is the bottleneck — check iostat and PCIe traffic. (2) NCCL collectives are eating time — see NCCL guide. (3) Python overhead in the dispatch loop — enable CUDA Graphs. (4) Optimizer step is serial — try fused optimizers (Apex, Liger). (5) Suboptimal kernel selection — profile with Nsight Compute.

Q: How do I handle multi-tenant GPU sharing?

Three options. (1) MIG (Multi-Instance GPU) on H100/H200 — partitions a GPU into 7 hardware slices with isolated SMs and memory. Strict isolation, no NVLink. (2) Time-sharing with MPS — multiple processes share contexts, lower isolation. (3) Pod-per-GPU with the NVIDIA device plugin — simplest, no sharing. Production multi-tenant inference typically uses option 3 for simplicity; option 1 for cost-optimized eval workloads.

Q: What's the deal with confidential computing on H100?

H100 supports Confidential Compute (CC) mode — encrypted memory, attestation, isolation. Performance cost: 5-10%. Useful for serving customer data under strict compliance regimes. Most workloads don't need it; some regulated industries (healthcare, defense) require it.

Q: When does TPU v5p beat H100?

Inside Google Cloud, with JAX, on training workloads that fit TPU sharding patterns (model parallelism via mesh primitives). TPU v5p delivers ~$15-18/chip-hour reserved versus ~$20-25/H100-hour reserved on GCP A3, with similar throughput per chip. Outside Google Cloud — never; TPUs aren't sold separately. The migration cost from PyTorch+CUDA to JAX+XLA is rarely worth the price savings.

Q: How does Grace+Blackwell (GB200) compare to x86+B200?

GB200 has 480 GB of LPDDR5X system memory per Grace plus 192 GB HBM per B200, with NVLink-C2C between them at 900 GB/s. Practical effect: optimizer states and checkpoints offload to Grace memory at NVLink speed, enabling larger models or smaller TP groups. Cost: 30-40% premium over x86+B200. Worth it for memory-constrained training; not for inference.

Q: Is there a cost-optimized GPU for embedding workloads?

L40S (48 GB HBM, 91 TFLOPs FP16, 350W, ~$8k) is the sweet spot. Throughput on BERT-large or ColBERT-style embedding generation is 60-70% of H100 at 30% of the price. For pure embedding services at scale, L40S clusters beat H100 on $/embedding by ~2×.

Q: How do AWS Trainium and Inferentia compare to H100 in 2026?

Trainium2 delivers ~$3-4/chip-hour for training-only workloads. Inferentia2 delivers ~$1-1.50/chip-hour for serving. Both require AWS Neuron SDK (a CUDA alternative) — software ecosystem lags PyTorch+CUDA by 6-12 months. For AWS-native workloads with stable architectures, they're 30-50% cheaper than H100/H200 on EC2. For novel research or multi-cloud, NVIDIA wins.

Q: What's the right SKU for serving a 405B dense model?

Today: 8× H100 80 GB (TP=8) with FP8 weights barely fits, KV cache extremely tight; 4× B200 (TP=4) with FP8 fits comfortably. Realistic production: 8× B200 (TP=8) for headroom and per-replica throughput, or 16× H100 (TP=8 × 2 replicas) for cost. GB200 NVL72 wins if you can absorb the full rack — TP=8 leaves 9 replicas per rack.

Table of contents

Key takeaways

Mental model: GPU architecture in one minute

Family tree: A100 → Hopper → Blackwell → Rubin

Hopper (H100, H200)

Why Hopper mattered

H100 — the workhorse

H200 — Hopper for inference

Hopper-specific software innovations

H100 NVL — the inference-optimized variant

Blackwell (B100, B200, GB200)

B200 — the headline

B100 — the lower-power sibling

GB200 NVL72 — rack-scale

Why Blackwell mattered

Blackwell HGX vs DGX

B200 in production

Spec sheet comparison

Compute throughput claim corrections

HBM: bandwidth and capacity

Capacity

Bandwidth

Why bandwidth matters more for inference

Tensor cores: FP16, FP8, FP4

FP16 / BF16

FP8

FP4

What about INT8 / INT4?

NVLink and node topology

Versions

Topology within a node

Why NVLink matters

Power and thermals

The Rubin family (2026–2027)

Cooling and power considerations

Power consumption progression

Rack density

Power efficiency (perf/watt)

Liquid cooling specifics

Pricing and availability

Lease vs buy economics

Cloud GPU options compared

Workload fit: training vs inference

Training: Blackwell wins

Inference: depends on context

Inference on Blackwell with FP4

Picking the right SKU

Capacity planning examples

Example 1: serving Llama-3 70B for 100 concurrent users at 32k context

Example 2: training a 7B from scratch in 14 days

Example 3: training a 405B model

Capacity planning for inference clusters

Capacity planning for training clusters

Workload-specific GPU selection deep dive

LLM inference, single user

LLM inference, multi-tenant chat

LLM inference, long context

LLM training, fine-tuning

LLM training, frontier-scale pre-training

Image/video generation (diffusion models)

Embeddings / classification

Reinforcement learning training

Edge inference

Power and cooling: from air to liquid

Air cooling limits

Liquid cooling enables density

Cooling infrastructure cost

Power supply implications

What this means for cloud customers

Heat reuse and sustainability

NCCL and GPU-to-GPU communication

What NCCL does

Topology-aware algorithms

Performance per generation

Tuning matters

Alternative communication libraries

Hopper-to-Blackwell migration tips

Software compatibility

Performance differences

Workloads that don't benefit much