A GPU cluster can have every cable connected and still be unready for training. The useful engineering question is not simply “NDR or XDR?” It is whether the port map, subnet management, routing policy, GPU-to-NIC alignment and failure behavior describe the same system.
This focused AI infrastructure installment develops a hypothetical 64-server, 512-GPU InfiniBand design, then turns it into a commissioning workflow. Calculations below are theoretical capacity accounting, executed with Python—not benchmark results. The example assumes eight GPUs and eight independent 400Gb/s network ports per server; it is not a claim that every eight-GPU server has that topology.
1. Start with the generation, model and actual port map
NVIDIA Quantum-2 supports 400Gb/s per port, with a configuration of 64 such ports per switch.[1] The QM9700/QM9790/QM9701 hardware manual identifies 32 physical OSFP interfaces and 64 NDR ports, and specifies 51.2Tb/s aggregate bidirectional switch throughput.[8] Do not interpret that headline as bandwidth available to one server, one collective operation or one direction across a rack boundary.
For Quantum-3, the current XDR manual describes the Quantum-X800 Q3400-RA as a 4U system with 144 ports at 800Gb/s across 72 OSFP cages.[2] It describes the 2U Q3200 as two independent switches in one enclosure, each offering 36 ports at 800Gb/s.[2] That is a material topology distinction: do not model the Q3200 as one fully interconnected 72-port switch just because the enclosure total is 72.
The NDR ordering table lists the documented QM9700 and QM9790 variants as “Mass Production.”[7] The XDR source establishes documented hardware configurations, not a verified delivery date for your region or bill of materials.[2] Before approving an XDR purchase, require the supplier to confirm orderable part numbers, delivery, firmware support and the exact host adapter/cable combination. This article does not infer immediate availability from a product page or call any product “latest.”
Logical links are not physical cable assemblies
Build the connectivity schedule around logical switch port, physical cage, channel within the cage, remote port and cable assembly. NDR port splitting can expose up to 128 NDR200 ports, so the intended operating profile also matters.[8] A row saying only “OSFP port 12 to server 12” is not enough to approve installation.
Record module type, breakout arrangement, length, polarity, airflow-compatible hardware and the supported rate at both ends. Count logical links first; derive transceiver, harness and spare quantities from the validated channel map. Two logical links that share a module may have a common maintenance failure domain. Label that dependency rather than presenting both as physically independent redundancy.
2. What the subnet manager owns—and what it does not
The subnet manager (SM) discovers and configures the InfiniBand fabric, including routing, partitioning and quality-of-service settings.[4] OpenSM initializes the subnet and attaches to a particular local InfiniBand port; it does not automatically manage every separate fabric connected to the host.[6] In the worked example, eight independent rail subnets therefore require explicit management coverage for all eight, not merely an SM process somewhere in the cluster.
My recommended management design is a documented primary/standby arrangement per subnet, with instances on separate failure domains and equivalent approved policy. Specify GUID binding, priority, configuration distribution, monitoring and failover ownership. Validate election behavior against the selected manager release rather than assuming that two enabled processes constitute high availability. Keep out-of-band access independent of the fabric being repaired.
Do not assume that buying an adaptive-routing-capable ASIC enables every platform feature through every management option. The cited MLNX-OS release explicitly lists limitations for its embedded SM, including adaptive routing, SHARP, congestion control and Dragonfly+ support.[4] Meanwhile, the Quantum-2 platform advertises adaptive routing, congestion control and SHARPv3 as platform capabilities.[1] These statements concern different layers, not a reason to ignore the software support matrix.
For procurement, request a signed-off combination of switch firmware, adapter firmware, host drivers, fabric manager, routing mode and any required feature licenses. Treat an older OpenSM manual here as a reference for management concepts and syntax—not an XDR qualification document. Changing manager implementation during commissioning should require a new baseline.
3. Rail alignment: start at the GPU, not the rack label
In this hypothetical design, each server contributes one network port to each of eight isolated rails. Rail 0 carries port 0 from every server; the same rule repeats through rail 7. The intended GPU-to-port relationship is an explicit design assumption that must be checked against the purchased server’s internal topology.
NCCL’s troubleshooting guidance separates GPU-to-GPU and GPU-to-NIC topology issues from networking diagnostics and runtime problems.[5] Follow that separation operationally: capture the GPU, NIC and CPU locality map before judging fabric performance. A correct cable label cannot prove that an application chose the intended adapter or that the host data path matches the reference architecture.
For every server, keep an inventory joining GPU identifier, PCIe location, adapter GUID, adapter port, rail, leaf and cage channel. Compare the discovered map with the approved design, and investigate exceptions rather than silently adapting scripts to whichever enumeration appeared after a reboot.
Independent rails are not Ethernet bonds. In this design there are no switch links joining one rail to another. Any desired application behavior after a missing rail must be validated with the chosen communication stack. Do not promise that a running training job survives a failed NIC simply because seven other NICs remain available.
4. Worked NDR sizing: 64 servers, eight rails
Hypothetical requirements: 64 servers, eight GPUs per server, eight 400Gb/s ports per server, no compute-fabric storage endpoints, and a symmetric two-leaf/two-spine topology on each rail. Each switch is modeled as a 64-port NDR device. The objective is equal nominal leaf downlink and uplink capacity in the healthy state, with spare spine ports for planned expansion.
On each rail, leaf A connects one port from servers 1–32; leaf B connects one port from servers 33–64. Each leaf spends its remaining 32 ports on uplinks: 16 to spine A and 16 to spine B. Each spine therefore terminates 32 links and retains 32 unused data ports. These are parallel physical links, not one link assumed to run faster than NDR.
| Item | Tool-calculated result |
|---|---|
| GPUs | 64 × 8 = 512 |
| Host endpoints per rail | 64 |
| Leaf downlink capacity, one direction | 32 × 400Gb/s = 12.8Tb/s |
| Leaf uplink capacity, one direction | 32 × 400Gb/s = 12.8Tb/s |
| Endpoint injection per rail, one direction | 64 × 400Gb/s = 25.6Tb/s |
| Switches across eight rails | 8 × (2 + 2) = 32 |
| Host-facing logical links | 64 × 8 = 512 |
| Leaf–spine logical links | 8 × 2 × 32 = 512 |
| Total data logical links | 1,024 |
| Server port-rate sum, one direction | 8 × 400Gb/s ÷ 8 = 400GB/s |
| Cluster endpoint injection, one direction | 64 × 8 × 400Gb/s = 204.8Tb/s |
These sums describe nominal port capacity. They exclude protocol overhead and do not claim RDMA payload throughput, NCCL bus bandwidth or delivered application bandwidth. The 204.8Tb/s figure is an endpoint injection sum, not a measured cluster bisection result. Likewise, the server’s 400GB/s sum is not a guarantee that host buses, GPU access paths or software can sustain it.
No aggregate bidirectional NVLink number is used in this comparison. Keep scale-up performance and scale-out port accounting on separate worksheets unless the directionality, units and transfer path are explicitly reconciled.
Capacity balance is not a congestion guarantee
Equal leaf downlink and uplink sums remove one obvious structural oversubscription point. They do not demonstrate even path use, eliminate incast or prove a collective will reach a target. OpenSM documents separate routing engines, including fat-tree routing for symmetric fat-tree layouts, with topology-specific constraints.[6] Approve routing for the actual topology and then test concurrent traffic, rather than translating “1:1” into “congestion-free for every job.”
For this model, approve the symmetric topology first, select a supported routing configuration, and preserve the discovered topology and forwarding state with each benchmark record. If cabling changes, require topology validation before comparing performance with the original baseline.
5. Failure domains and alternative designs
Losing one spine leaves each leaf with 16 surviving 400Gb/s uplinks: 6.4Tb/s, versus 12.8Tb/s of host-facing capacity. That is a calculated 2:1 downlink-to-uplink ratio for the affected rail. It does not mean every job becomes exactly twice as slow; traffic locality, routing convergence and application behavior remain part of the test.
Losing a leaf removes that rail’s network attachment for 32 servers in this design. A shared power, management or cable-path event may affect more than the logical diagram suggests. Losing the active SM is a different test from losing a forwarding switch: record management recovery and application behavior separately. Never write “redundant” without specifying which failure the claim covers.
A smaller alternative is one 64-port NDR switch per rail. It uses fewer switches and removes the leaf–spine cable layer for these 64 endpoints, but leaves no spare data ports and places each entire rail behind one switch failure domain. Choose it only if that growth and outage model is acceptable.
An XDR alternative must be redesigned around its actual radix, host adapters and cable mapping; doubling the nominal link rate is not a procurement plan. The documented Q3400 offers 144 800Gb/s ports, while the Q3200 divides its enclosure ports across independent switches.[2] Compare lifecycle cost, support, spare strategy, host bottlenecks and expansion steps—not just nominal terabits. Ethernet/RoCE is another architecture option, but deserves its own congestion-control and operations design rather than copied InfiniBand settings.
6. Partitions: verify communication policy explicitly
An InfiniBand partition defines which end ports may communicate; the partition manager assigns PKeys to channel-adapter ports, and invalid PKeys are discarded at the documented enforcement points.[4] OpenSM creates a default partition even if no partition file is available, and its manual documents both full and limited membership.[6] Therefore, the existence of a named training partition is not proof that unintended communication paths are absent.
For this example, maintain separate approved membership lists for training and infrastructure access, then test both allowed and denied pairs. Do not treat a PKey as encryption or as a complete hostile-tenant security architecture. Also do not assume that naming partitions reserves bandwidth: document and validate the associated QoS policy separately.
The following is an unexecuted template, based on the documented OpenSM partition syntax; placeholders are deliberately non-deployable.[6]
# Illustrative structure only. Replace placeholders with approved port GUIDs.
Training=0x1234,defmember=limited:
<TRAINING_PORT_GUID_A>=full,
<TRAINING_PORT_GUID_B>=full;
This is not a complete production partition file. It does not define your default-partition policy, manager reachability, storage membership or IP-over-InfiniBand multicast requirements. Review those explicitly, distribute the same approved configuration to standby management, and conduct negative tests from endpoints outside the permitted group. Retain out-of-band access before changing a policy that could cut off in-band administration.
7. Commissioning workflow: prove each layer separately
Stage A: freeze the physical and software inventory
Require the delivered switch and adapter part numbers, firmware versions, driver stack, manager release, cable schedule and GPU/NIC topology export. Match every expected endpoint to a discovered GUID. Record negotiated link rate and width against the intended channel profile; investigate a lower-than-designed link before running cluster-wide workloads.
The following unexecuted diagnostic examples are starting points, not captured output or a validated command sequence for your installed release. Check local help, package availability, privileges and adapter-selection options first. Discovery output can contain identifying information; keep raw captures private.
ibstat
ibv_devinfo
ibnetdiscover
nvidia-smi topo -m
On a multi-adapter management host, explicitly select the intended rail using the installed tool’s supported options. A successful discovery on one subnet does not validate the other seven. Archive results with a timestamp and the design revision so later changes are attributable.
Stage B: prove management and policy
Check that every subnet has the expected active manager, that standby coverage is healthy, and that routing and partition policies match the approved revision. OpenSM documents “SUBNET UP” as an initialization success indication in its logs.[6] Treat that as a bring-up milestone, not a performance certificate or proof of tenant isolation.
Schedule a controlled manager failover on one rail. Record election/recovery timing, endpoint state, policy consistency and the behavior of an agreed test workload. Restore the primary arrangement and repeat validation. A quiet cluster and a saturated cluster should be separate test cases.
Stage C: progress from one path to collective traffic
Begin with a single approved endpoint pair, then test multiple simultaneous pairs, then representative collectives. Include same-leaf and cross-leaf pairs, each rail individually, and the intended multi-rail application configuration. NVIDIA’s NCCL troubleshooting structure specifically calls for low-level fabric checks and latency/bandwidth investigation before performance tuning.[5]
Use the same message sizes, duration, software build, placement and warm-up procedure across comparisons. Confirm that the intended RDMA transport and adapters were selected; do not infer that from successful job startup. Record both the application result and relevant link counters. A clean single-pair test is necessary evidence, but not sufficient evidence for a busy cluster.
Stage D: define pass/fail before the maintenance window
Agree numerical performance and recovery thresholds with the system supplier and workload owner using a qualified reference configuration or pilot baseline. This article supplies no invented acceptance percentage. Require reproducibility and correct results, not only the best throughput sample.
An acceptance record should state test identity, host placement, rail selection, configuration revision, traffic pattern, observed result, error-counter deltas and pass/fail rationale. For failure tests, add the exact injected fault, expected scope, actual scope and restoration evidence. Drain production workloads and use an approved maintenance procedure before pulling a cable or disabling a component.
8. Troubleshooting by symptom
| Symptom | First investigation | Evidence to retain |
|---|---|---|
| One rail absent from discovery | Adapter binding, physical path, SM coverage | GUID inventory and manager logs |
| Links up, cross-leaf traffic weak | Uplink mapping, routing, negotiated rates | Topology, forwarding state, per-link utilization |
| One server consistently slow | GPU/NIC locality, host software, adapter health | Topology export and controlled pair results |
| Named partition exists but isolation fails | Default partition and actual endpoint membership | Allowed/denied pair test matrix |
| Pair test passes but collectives stall | Concurrent load, transport choice, runtime | Application logs, placement and fabric counters |
| Fault affects several rails | Shared cage, cable path, power or management dependency | Physical dependency map and event timeline |
Use this table as a triage plan, not a diagnosis inferred from one counter. Change one variable at a time, preserve the previous configuration and rerun the same test. Avoid simultaneous firmware, routing and application-tuning changes: even an improvement becomes difficult to explain or reproduce.
Practical takeaways
Approve a port map before a cable order. Qualify the management stack before enabling advanced features. Budget host links and fabric links separately, and distinguish logical links from physical assemblies. Test failure scope and recovery rather than assuming spare capacity provides application continuity.
For the worked design, the healthy-state leaf budget balances at 12.8Tb/s in each direction of the downlink/uplink comparison; one spine failure halves the surviving uplink capacity on that rail. Those are planning calculations. Only staged, recorded acceptance tests can establish whether the actual system meets the workload requirement.
Continue with the AI Infrastructure and Automation hub and Data Center Networking hub. Related earlier coverage: AI cluster network telemetry and out-of-band management design. The next design step is to turn the approved topology into a complete leaf/spine port budget, expansion plan and cable schedule.
Sources
- [1] https://www.nvidia.com/en-us/networking/quantum2
- [2] https://networking-docs.nvidia.com/xdrswitcheshw/introduction
- [4] https://networking-docs.nvidia.com/mlnxosum/3126200lts/subnet-manager
- [5] https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html
- [6] https://networking-docs.nvidia.com/mlnxofedswum/586042lts/opensm
- [7] https://networking-docs.nvidia.com/qm97x0hw
- [8] https://networking-docs.nvidia.com/qm97x0hw/introduction
Post a Comment