A slow AllReduce is not automatically a slow switch. Before buying faster NICs or copying NCCL tuning variables, isolate whether the problem follows a GPU pair, a server, a rail, a rack boundary, or the application. This guide provides a reusable troubleshooting matrix, a staged test plan and a worked bandwidth example for an AI training cluster.
Scope: NVIDIA Collective Communications Library (NCCL), GPU servers using InfiniBand or RDMA over Converged Ethernet (RoCE), and the boundary between local GPU communication and scale-out networking. All designs, timing inputs and acceptance policies below are hypothetical. Commands are unexecuted examples for an authorized maintenance allocation—not reported benchmark runs. Documentation was checked on September 11, 2026; match settings to your installed NCCL, CUDA, driver and platform versions rather than assuming rolling documentation describes your deployment.
Quick answer: find the smallest failing boundary
NVIDIA separates NCCL troubleshooting into GPU paths, networking, runtime/MPI, logging and performance tuning, rather than treating environment-variable changes as the first step.[1]
Use this diagnostic order:
- Confirm correctness and launch behavior before interpreting throughput.
- Test GPU pairs and the full local GPU group on each server.
- Test a selected NIC pair using host memory, then supported GPU-memory RDMA.
- Run two-node NCCL with controlled placement; repeat across rails and rack boundaries.
- Expand to the intended allocation and compare with the application's actual communication pattern.
- Change one tuning setting only after the lower layers have evidence of health.
NVIDIA specifically recommends separating low-level GPU and network performance from NCCL configuration problems before tuning NCCL.[7] The sequence here is an operational worksheet, not a promise that every fault has only one cause.
The reusable NCCL troubleshooting matrix
Treat the “next test” column as a way to discriminate causes, not as an automatic diagnosis.
| Symptom | Working hypothesis | Next controlled test | Evidence to retain |
|---|---|---|---|
| Initialization hangs before any timings | Launcher, bootstrap reachability, resource limits | MPI launch/reduction check; NCCL initialization logs | First failing rank, interface choice, actual job limits |
| One server is slow even without external traffic | Local topology, GPU path or placement | Local GPU pair tests and full-node AllReduce | GPU-pair map, CPU binding, version inventory |
| Host-memory RDMA is healthy, GPU-memory RDMA is not | GPU-to-NIC path or GPUDirect setup | Same endpoints and NIC, change memory type only | DMA-BUF/peer-memory mode, PCIe locality |
| One isolated rail is consistently worse | Rail-specific endpoint, cable, route or link issue | Repeat that rail across several node pairs | Port rates and counter deltas during each run |
| Within-rack is good, cross-rack is poor | Uplink/path capacity, placement or congestion | Same rank count and message sweep across boundary | Switch path, competing traffic, latency variation |
| Microbenchmark is healthy, training is slow | Different communication pattern or work imbalance | Application trace with representative rank placement | Collective sizes, arrival timing, compute/input stalls |
| Large buffers look good, small buffers do not | Latency-sensitive behavior hidden by peak bandwidth | Small-message latency sweep, fixed launch method | Time units, size curve, run-to-run variation |
The layer choices reflect NVIDIA's GPU, network and runtime diagnostic guidance; the matrix itself is an original triage aid.[5][6][9]
Make “slow” a reproducible comparison
Create a run manifest before changing anything. Record the GPU model and form factor, server model, GPU count, actual NVLink/NVSwitch topology, NIC model and port count, negotiated rates, switch paths, firmware, driver, CUDA, NCCL, network plugin, MPI and container image. Also record the scheduler allocation, CPU/GPU binding, environment overrides, message sweep and background load.
Do not compare an eight-GPU NVSwitch server with a collection of PCIe cards merely because the GPU product family looks similar. NVIDIA's diagnostic guidance distinguishes NVLink, PCIe, GPU-to-NIC paths and the topology exposed to NCCL through /sys.[5] Inventory the real system rather than assuming a product name specifies the wiring.
Recommended baseline policy: retain the same size sweep, datatype, reduction operation, placement and correctness mode across repeats. Save all results, not just the fastest line. Use the same logging level on both sides of a comparison, then repeat final performance measurements with normal production logging. This keeps the experiment interpretable without pretending that a single universal performance threshold applies to every cluster.
Understand algbw and busbw before calling the network slow
In nccl-tests, algorithm bandwidth is message size divided by operation time: algbw = S / t.[3] For AllReduce, the reported bus-bandwidth normalization is busbw = algbw × 2 × (n − 1) / n, where n is the collective's rank count.[3]
That normalization is not a switch-port counter. NVIDIA describes it as a way to relate collective performance to the hardware bottleneck, which can be NVLink, PCIe, an inter-socket link or the network.[3] In a hierarchical multi-GPU server, do not interpret every normalized byte as a byte that crossed a particular external NIC. Confirm the actual communication path through logs and correlated interface counters.
Worked example: 64 ranks, not a measured benchmark
Assume eight servers with eight GPUs each, one GPU per rank, and a 256 MiB AllReduce buffer per rank. Assume the collective completes in 20 milliseconds purely for arithmetic illustration. The following values were calculated with Python; none is a measured result.
| Quantity | Calculation | Result |
|---|---|---|
| Ranks | 8 servers × 8 GPUs | 64 |
| Buffer per rank | 256 × 1,048,576 bytes | 268,435,456 bytes |
| Algorithm bandwidth | 268,435,456 / 0.020 / 1,000,000,000 | 13.422 GB/s |
| AllReduce normalization factor | 2 × 63 / 64 | 1.96875 |
| Normalized bus bandwidth | 13.4217728 × 1.96875 | 26.424 GB/s |
| Assumed 400 Gb/s NIC rate, converted | 400 / 8 | 50 GB/s, theoretical single direction |
The formulas come from the NCCL test documentation; the input size, elapsed time and hardware rate are selected example inputs.[3] Read the time-unit header printed by your own binary and convert it explicitly. Do not paste an unlabeled time-column number into a spreadsheet expecting seconds.
For an intentionally simplified flat model with one independent 50 GB/s path per rank and no overhead, the same formula gives about 10.570 ms. That is an idealized model result, not an acceptance target for the eight-server topology. Local aggregation, algorithm choice, protocol overhead and shared paths make the mapping between this toy model and a real system more complicated.
The useful outcome is not “the cluster is half speed.” It is a precise question: which path limits the observed operation, and what measured baseline exists for that exact allocation? Never compare an aggregate bidirectional NVLink headline directly with the single-direction NIC rate in this table.
Step 1: prove local GPU communication first
NVIDIA recommends topology inspection and GPU bandwidth testing to identify local communication issues before concluding that NCCL needs tuning.[7] Use the following read-only inventory examples on allocated GPU nodes:
# Unexecuted examples; availability depends on the installed tools.
nvidia-smi
nvidia-smi topo -m
nvidia-smi topo -p2p n
nvidia-smi topo -p2p p
lscpu --extended=CPU,NODE,SOCKET,CORE
Peer-access status is not a bandwidth measurement: NVIDIA recommends nvbandwidth to measure GPU-to-GPU bandwidth, while its troubleshooting guidance also describes correctness checks using CUDA's simpleP2P sample.[5] Test relevant pairs within and across the server's local topology boundaries, then run the complete local collective.
The NCCL test suite checks correctness as well as performance and documents a single-node eight-GPU invocation.[2] A bounded adaptation is:
# Unexecuted: reserve a node with eight accessible GPUs first.
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
The sweep starts at eight bytes and doubles to 128 MiB; confirm memory availability and the test version's options before choosing a larger limit.[2] Keep the correctness output with the performance output. An allocation failure or validation error is not a valid slow-bandwidth sample.
If the local result is poor, inspect GPU/NIC/CPU locality and the actual launcher binding before touching switches. NVIDIA notes that ranks and host memory should generally be placed near their GPU and NIC, and that inherited affinity and container restrictions affect usable CPUs.[7]
Safety boundary: ACS and IOMMU can affect GPU peer traffic, but the requirements differ between bare metal and virtual machines.[5] Do not apply blanket PCI configuration writes or disable isolation controls from a troubleshooting blog. Escalate platform-specific changes to the server vendor and virtualization/security owners with a rollback plan.
Step 2: separate the NIC path from the GPU-memory path
Start with link state, rate and link layer. NVIDIA documents ibstat and ibstatus for this purpose, then ib_write_bw and ib_write_lat for bandwidth and latency testing.[6]
# Unexecuted read-only inventory examples.
ibstat
ibstatus
rdma statistic
Choose an approved pair of nodes and one identified adapter. Replace the device and peer placeholders with values from your private run manifest; never publish that manifest unchanged.
# Unexecuted host-memory example, terminal on the server node:
ib_write_bw -d <HCA_DEVICE> -a
# Unexecuted matching client example:
ib_write_bw -d <HCA_DEVICE> <PEER_ADDRESS> -a
These are diagnostic workloads, not passive checks: run them in an allocation where fabric load is permitted. Use matching perftest versions/options on both endpoints and validate the site's addressing requirements. For RoCE, confirm the appropriate GID/addressing setup rather than assuming an InfiniBand example is universally sufficient.
Next compare supported GPU-memory mode against the same host-memory test. NVIDIA lists --use_cuda and, for supported builds and platforms, --use_cuda_dmabuf; it explicitly advises checking the installed ib_write_bw --help for available GPU-memory options.[6] Configure the corresponding mode on both peers and keep the GPU-to-NIC locality controlled.
GPUDirect RDMA may use the nvidia-peermem path, while supported DMA-BUF configurations can avoid requiring that module.[5] Therefore, “module absent” alone is not a diagnosis. Establish the intended mode and its prerequisites from the installed driver, kernel, GPU and NIC combination.
If host memory performs well but GPU memory does not, focus the next investigation on the GPU-to-NIC path. If both are poor, investigate endpoint/link/fabric conditions before tuning the collective. Neither comparison alone proves a particular cable or driver is defective.
Step 3: prove NCCL selected the intended transport
NCCL's logging controls include NCCL_DEBUG, subsystem filtering, and per-process files using %h for hostname and %p for PID.[8] A focused diagnostic setup is:
# Unexecuted: use a fresh private log directory for every run.
mkdir -p "$HOME/nccl-diagnostic-run"
chmod 700 "$HOME/nccl-diagnostic-run"
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,ENV
export NCCL_DEBUG_FILE="$HOME/nccl-diagnostic-run/nccl.%h.%p.log"
Ensure those variables reach every rank and the directory exists on every participating node. Collect the logs privately: they can contain hostnames, device identifiers and other infrastructure details. NVIDIA recommends unique filenames across processes to avoid log corruption.[8]
Read transport selection and topology together. NCCL still uses TCP connections to exchange connection information even when the data path uses InfiniBand/RDMA, so the appearance of an IP interface is not itself evidence of socket fallback.[6] Look for the selected network transport and devices in the NET logs, then correlate traffic with the intended NICs.[8]
NCCL_SOCKET_IFNAME filters IP interfaces; prefix matching and exact matching have different syntax.[4] It is not a substitute for selecting RDMA HCAs. If a controlled test requires adapter selection, consult the installed version's NCCL_IB_HCA documentation and verify the resulting device/port selection in logs.[4]
Avoid a global “known fast” environment file. NVIDIA warns that retained debug/tuning settings can cause suboptimal behavior, crashes or hangs.[4] Keep an override list in the experiment manifest and remove temporary overrides before the final baseline.
Step 4: expand through rails and rack boundaries
Hypothetical topology and capacity budget
Assume eight conventional GPU servers, each containing a local eight-GPU NVLink/NVSwitch domain and eight independent 400 Gb/s NIC ports. Assume one port from each server connects to each of eight isolated InfiniBand rails. This is an abstract diagnostic design, not a statement about every GPU server or a vendor-qualified bill of materials.
The endpoint budget is 64 NIC ports, with eight host-facing ports per rail. The per-server NIC sum is 400 GB/s theoretical single-direction line rate. Switch uplinks, usable logical port counts, breakout rules, cable choices and reserve capacity require their own model-specific design review; these endpoint counts are not a complete switch BOM.
Run the same two-node collective on a selected rail, then repeat across rails and node pairs. Follow with all intended rails enabled. Finally, expand from two nodes to the full allocation, preserving per-node process placement and the message sweep. NVIDIA documents both rail-sensitive NCCL_CROSS_NIC behavior and the possibility that forcing cross-NIC traffic can create congestion on some networks.[7]
For multi-node nccl-tests, the suite must be built with MPI support; the total GPU-rank count is processes × threads × GPUs per thread.[2] The following is an Open MPI-style template, not a scheduler-independent recipe:
# Unexecuted: inside an authorized two-node, eight-GPU-per-node allocation.
# Build with MPI=1 first; propagate logging variables to remote ranks.
mpirun -np 16 -N 8 \
-x NCCL_DEBUG -x NCCL_DEBUG_SUBSYS -x NCCL_DEBUG_FILE \
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 1
Verify GPU assignment and CPU binding rather than assuming the launcher selected good locality.[7] Before blaming NCCL initialization, NVIDIA recommends proving MPI can launch processes and complete a simple reduction.[9]
Failure domains and alternative choices
In this hypothetical design, a server outage removes eight GPUs. One complete rail outage removes one eighth—12.5%—of the provisioned external endpoint rate per server. That arithmetic says nothing about graceful job survival: define separately whether the deployed stack aborts, retries or requires a fresh allocation after a fabric failure. Do not promise transparent failover based on port redundancy alone.
An alternative with fewer NICs per server reduces endpoint and cabling demand, but gives more GPUs a shared external budget. Choose between them using representative workload tests, not the largest single-buffer score. For a multi-node NVLink platform, the diagnosis must also follow the actual NVLink domain and associated services rather than assuming every server-to-server transfer crosses InfiniBand; NVIDIA documents separate multi-node NVLink checks.[7]
Step 5: investigate congestion without random tuning
For InfiniBand, NVIDIA recommends checking Subnet Manager availability, link/error counters and fabric-wide diagnostics such as ibdiagnet when a rail behaves differently.[6] Collect counter deltas over the workload interval; an old cumulative error count without timing context is a weak basis for a change.
For RoCE, correlate NIC and switch queue drops, priority flow control (PFC), explicit congestion notification (ECN) and congestion notification packet (CNP) indicators with bandwidth instability or growing latency tails.[6] These signals help locate congestion behavior; the worksheet does not prescribe universal buffer, threshold or traffic-class values.
A particularly dangerous shortcut is importing historical RoCE configuration. NVIDIA documents dynamic GID selection from NCCL 2.21 onward and advises not setting NCCL_IB_GID_INDEX on those releases.[6] Check the installed release and remove obsolete overrides deliberately, rather than replacing one unexplained setting with another.
Only after the lower-layer tests are healthy should you consider controlled NCCL changes such as path-diversity or rail-selection settings. NVIDIA cautions that tuning gains for one benchmark can hurt other cases.[7] Require improvement across the relevant message distribution and the real application, and preserve an immediate rollback to defaults or the previous validated configuration.
Deployment and acceptance checklist
Use this as a durable handover asset. Fill in actual pass criteria before the supplier or operations team runs acceptance:
- [ ] Hardware, firmware, libraries, container and test-suite revision are recorded.
- [ ] Physical ports, logical rails, GPU locality and scheduler placement match the approved design.
- [ ] Correctness passes for local and multi-node tests; invalid runs are excluded, not averaged.
- [ ] Host-memory and GPU-memory network tests have comparable endpoint and placement records.
- [ ] NCCL logs confirm the intended transport, devices and topology.
- [ ] Small-message time and large-message bandwidth are both retained with explicit units.
- [ ] Repeated runs cover node pairs, rails, rack boundaries and the intended full allocation.
- [ ] Counter deltas and latency variation are captured under quiet and agreed competing-load conditions.
- [ ] Representative training is tested with the intended framework, bucket sizes and placement.
- [ ] Maintenance failure drills have explicit abort/recovery criteria and approved scope.
- [ ] Temporary debug/tuning settings are removed; the final baseline is repeated.
- [ ] Raw evidence is retained privately and the public summary contains no infrastructure identifiers.
For acceptance numbers, agree a hardware-specific reference and an allowed regression band for each test class. Do not invent a universal percentage of NIC headline rate: the relevant bottleneck and normalization must be understood first.
Practical takeaways
The durable fix for slow AllReduce is an evidence chain: local GPU health, GPU-to-NIC health, selected transport, fabric boundaries and finally application behavior. Use busbw as a normalized collective metric, not a literal NIC utilization reading. Keep one variable per experiment, record the smallest failing boundary, and demand correctness alongside performance.
Continue with the earlier InfiniBand NDR/XDR design and acceptance installment and the GPU cluster network sizing calculator. The AI Infrastructure hub and Data Center hub collect related design and troubleshooting material.
Sources
- [1] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html
- [2] https://github.com/NVIDIA/nccl-tests
- [3] https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md
- [4] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
- [5] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/gpu_troubleshooting.html
- [6] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html
- [7] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/performance_and_tuning.html
- [8] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/logging.html
- [9] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/runtime_and_mpi_issues.html
Post a Comment