A GPU cluster is not ready for handover just because every server boots and one AllReduce finishes. The buyer needs a repeatable answer to a narrower question: does this exact hardware and software build deliver correct, stable communication across every intended failure domain, and can operations reproduce the evidence?
This AI GPU cluster acceptance checklist turns that question into a staged test plan for a hypothetical 256-GPU deployment. The durable assets are a commissioning matrix, coverage worksheet, evidence manifest and sign-off gates. It is a handover guide, not another list of NCCL tuning variables. All sizing figures are calculated examples; no GPU benchmarks were executed for this article.
1. Define the acceptance contract before running benchmarks
Write down the purchased topology and workload before selecting a target bandwidth. Our hypothetical design has 32 servers, eight GPUs per server and eight 400 Gb/s fabric-facing NIC ports per server, arranged as eight rails. Assume a vendor-supported GPU-to-NIC mapping and a leaf/spine fabric. These are planning inputs, not a claim about a particular shipping server or a recommendation that every GPU needs its own NIC.
Specify the GPU generation, form factor, memory size, intra-node interconnect, NIC model, negotiated link rate, switch topology and software bill of materials. Do not treat an eight-GPU PCIe server, an NVSwitch-connected server and a multi-tray NVLink system as interchangeable. Record which paths should stay inside the scale-up domain and which must cross the scale-out network.
For procurement, require the supplier to propose a reproducible reference result on the same configuration, not a slide showing another GPU generation. Attach the exact collective, message sizes, rank placement, firmware, library versions and permitted power settings. Agree on performance tolerances and excluded conditions before delivery. Otherwise, a test can be technically successful while answering the wrong contractual question.
Use separate gates for correctness, coverage, performance and recovery. A fast result with validation disabled is not a correctness pass. A successful local test is not proof that the fabric works. A graceful restart is not proof that an existing training job survives a link failure. Keep those distinctions visible on the sign-off sheet.
2. Build a versioned evidence manifest
For each run, save a manifest containing the run identifier, timestamps, scheduler allocation, server inventory, GPU identities, NIC-to-GPU mapping, physical rail map, software versions and executable checksums. Include container image digests when containers are used. Store the actual command, environment variables and standard output/error alongside telemetry from the same interval.
Keep private hostnames, serial numbers and management addresses in the operational archive, not in public reports. A public summary can use neutral identifiers. The team doing acceptance still needs a private translation back to real hardware; anonymization should not destroy fault isolation.
Record the intended transport and the observed transport separately. Similarly, record requested and observed GPU counts. A launcher returning zero does not establish that every purchased device participated. Make the manifest a required input to review, rather than documentation assembled after a suspicious result.
Freeze the reference software build during acceptance. If a firmware upgrade or library change is necessary, create a new baseline identifier and repeat affected tests. Do not silently mix results from different builds into one average. A later regression investigation should be able to reconstruct both the passing configuration and the changed configuration.
3. Gate A: establish local health before blaming the fabric
NVIDIA DCGM Diagnostics is intended to assess readiness, detect software and integration problems, and exercise hardware-related checks; NVIDIA explicitly says it is not comprehensive hardware diagnosis or a replacement for field diagnostic tools.[2] Use it as one layer of evidence, not a universal warranty certificate.
DCGM exposes numbered diagnostic run levels and JSON output through dcgmi diag.[2] The following are unexecuted example commands for a drained, approved test system with the required DCGM components already installed:
# Examples only: confirm supported options on the installed release.
dcgmi diag --help
dcgmi diag -r 1 -j > dcgm-readiness.json
# Run only after maintenance approval and workload evacuation.
dcgmi diag -r 3 -j > dcgm-extended.json
Do not launch stress tests on shared production GPUs merely because the command is short. Agree on power/cooling readiness, monitoring, abort criteria and ownership first. Capture the process exit code and examine the diagnostic report, including skipped tests. An unsupported or skipped test is an evidence gap, not a pass.
A particularly important boundary: the NCCL plugin described in the current DCGM documentation runs single-node tests, not multi-node NCCL tests.[2] Its binaries and library prerequisites are separate, and the plugin can be skipped when its binary-path environment is not configured.[2] Therefore, a green DCGM summary must not replace the scale-out stages below.
For local interconnect measurements, nvbandwidth supports host/device and device/device copy tests and distinguishes copy-engine from kernel-copy methods.[3] Run the chosen test consistently across servers; do not compare results from different methods as though they measure the same path. Start with the installed tool's test inventory:
# Unexecuted examples; run inside the approved GPU allocation.
./nvbandwidth -l
./nvbandwidth -t device_to_device_memcpy_write_ce
The upstream tool documents that ordinary unidirectional matrix entries are measured individually, not all at once.[3] Consequently, a healthy pairwise matrix is valuable local coverage but not a demonstration of simultaneous node-wide throughput. Preserve the matrix rather than just its best entry.
4. Gate B: prove rank placement and transport on two nodes
The NCCL tests project checks performance and correctness, and multi-process/multi-node operation requires an MPI-enabled build.[1] Pin the source revision and build inputs. Do not fetch and build an unreviewed moving branch during a contractual acceptance run.
NCCL tests calculate the total participating rank count from processes, threads per process and GPUs per thread.[1] For the example below, the design intent is one process per GPU across two eight-GPU servers. The launcher must actually provide that placement; merely typing a process count does not select the right hosts.
# Unexecuted Open MPI-style example, inside an exclusive two-node allocation.
# Requires nccl-tests built with MPI=1 and validated GPU visibility/binding.
mpirun -np 16 -N 8 ./build/all_reduce_perf \
-b 8 -e 1G -f 2 -g 1 -w 5 -n 20 -c 1
Here, -g controls GPUs per thread, -w warmup iterations, -n timed iterations and -c correctness-check iterations in the documented interface.[1] Adapt the launcher to the site's scheduler integration. In particular, confirm local rank-to-GPU assignment rather than assuming container device visibility and MPI placement always agree.
Test a pair under the same leaf and a pair crossing the intended leaf/spine boundary. Repeat across planned rail mappings. Require the run log and fabric counters to demonstrate that traffic reached the intended network. A successful test using an unintended path should be filed as a configuration finding, not promoted to the fabric baseline.
Resist the urge to force many NCCL variables immediately. First establish what the supported default configuration selected. If a diagnostic override isolates a problem, record it as an experiment and restore the reference configuration before final acceptance. An unexplained collection of overrides is a poor handover artifact.
5. Coverage sizing: how much testing is enough?
The hypothetical inventory contains 32 × 8 = 256 GPUs. A directed local GPU-pair inventory contains 32 × 8 × 7 = 1,792 entries. That is a coverage count, not a promise that every topology exposes an equivalent direct link or that every tool reports the matrix identically.
There are 32 × 31 / 2 = 496 unordered server pairs. Exhaustively exercising every pair at every size and every rail may be expensive. For an initial smoke campaign, 16 disjoint server pairs cover all 32 servers once; testing eight corresponding rail paths per pair creates 128 planned pair/rail cases. This sampled coverage does not establish all-pairs connectivity or every spine path.
Add targeted pairs spanning each rack, leaf group and routing boundary. Rotate the disjoint pairing in later runs so the same two healthy neighbors cannot hide a broader fault. Use the physical topology to justify the sample, and explicitly list what was not covered. Never label a sampled test as exhaustive simply because every server appeared somewhere.
For a concrete scheduling worksheet, assume 12 selected campaign cases, three repeats and five minutes reserved per repeat: 36 runs and 180 minutes of serial test slots. Five minutes is an administrative allowance, not a predicted benchmark runtime. Add queueing, launch, initialization, telemetry collection, troubleshooting and longer soak tests separately. Parallelize only when shared-resource interference is either excluded or deliberately under test.
| Planning item | Calculated example | What it does not prove |
|---|---|---|
| Purchased GPUs | 256 | Correct discovery or placement |
| Directed local pair entries | 1,792 | Concurrent node bandwidth |
| Unordered server pairs | 496 | Actual routing-path diversity |
| Initial sampled pair/rail cases | 128 | Exhaustive fabric coverage |
| Repeated campaign runs | 36 | Sufficient thermal soak |
| Serial slots at assumed five minutes | 180 minutes | Actual completion time |
6. Interpret bandwidth without buying the wrong conclusion
Eight hypothetical 400 Gb/s NIC ports represent 400 GB/s of aggregate one-direction nominal link rate per server, with 50 GB/s per port, before protocol overhead and system limits. These figures are theoretical arithmetic, not measured payload rates. They do not establish PCIe capacity, GPU access efficiency, fabric bisection bandwidth or application performance.
NCCL tests define algorithm bandwidth as data size divided by time and apply a collective-specific correction for bus bandwidth; for AllReduce the factor is 2 × (ranks − 1) / ranks.[4] Treat busbw as a normalized collective metric, not a direct reading of one NIC's byte counter.
For a deliberately hypothetical 256-rank AllReduce processing a 1 GiB array in 50 ms, the calculations give 21.475 GB/s algorithm bandwidth and 42.782 GB/s bus bandwidth, using a correction factor of 1.9921875. Those are invented inputs to an executed calculation, not benchmark observations. This is why a spreadsheet must preserve message units, rank count, operation and elapsed-time units.
# Executable arithmetic example; no GPU or benchmark dependency.
ranks = 256
size_bytes = 2**30
seconds = 0.050 # hypothetical input, not measured
algbw_gbs = size_bytes / seconds / 1e9
busbw_gbs = algbw_gbs * 2 * (ranks - 1) / ranks
print(f"algbw={algbw_gbs:.3f} GB/s; busbw={busbw_gbs:.3f} GB/s")
Do not divide that busbw figure by the server's aggregate NIC figure and call the result utilization. The selected collective, topology and rank placement determine what a comparison means. Likewise, do not compare an aggregate bidirectional NVLink marketing number with a single-direction NIC rate. Preserve the measurement boundary before drawing conclusions.
The nvbandwidth documentation also warns through its measurement definitions that reported bidirectional results are per-direction under opposing traffic, not an aggregated sum.[3] Read the selected test's definition instead of inferring units from the word “bidirectional.”
7. Gate C: expand scale and introduce controlled contention
Use a progression from one server to two, a leaf group, cross-leaf groups and the complete allocation. Keep message sizes and software constant during the first scaling sweep. Then add the workload-relevant collectives and sizes. A training workload with significant AllGather or ReduceScatter deserves evidence for those operations, not only AllReduce.
Define performance floors against a qualified, topology-matched reference. For example, if the agreed reference for one exact case were 42 GB/s and the contract allowed a 10% reduction, the floor would be 37.8 GB/s. Both the reference and tolerance here are hypothetical. They are not NVIDIA guarantees or universal acceptance thresholds.
Evaluate repeated-run variation and the worst relevant placement, not only the best score. Keep small-message time separate from large-message throughput. Use a consistent reporting method and retain raw observations so reviewers can inspect outliers rather than accepting a polished average.
After isolation tests pass, introduce realistic overlap: checkpoint writes, data loading and representative compute activity. Label this as a different test condition. If overlap changes performance, compare timestamps across GPU, NIC, switch and storage evidence before declaring a network defect. The purpose is to identify the constrained shared resource, not to optimize an isolated microbenchmark at the expense of the actual job.
8. Gate D: test failure domains without promising transparent survival
Write expected outcomes before disabling anything. Separate path recovery, new-job admission, running-job behavior and checkpoint restart. For each drill specify the approved fault, blast radius, observer, abort condition, rollback and evidence required to return equipment to service.
| Approved drill | Observe | Suggested acceptance evidence |
|---|---|---|
| One selected fabric path unavailable | Alarms, routing/path changes, job outcome | Expected detection and recovery behavior; restored counters |
| One server unavailable | Scheduler exclusion and workload outcome | No new placement there; documented restart procedure |
| Planned control-service handover | Management continuity and application effect | Vendor-supported behavior matched to the runbook |
| Checkpoint restart on a fresh allocation | Data availability and resumed workload | Verified application state and measured recovery interval |
| Monitoring path interruption | Alert visibility and telemetry gaps | Explicit gap indication rather than false healthy status |
These are test design suggestions, not assertions that every platform supports hitless recovery. Use supplier-approved procedures for fabric control-plane transitions. Do not improvise power or cooling failures; qualified facilities personnel must own electrical and mechanical work. Re-run an agreed health subset after restoration and confirm the scheduler has not silently readmitted unhealthy resources.
9. A commissioning matrix you can put into the handover pack
| Gate | Minimum artifact | Hold acceptance when |
|---|---|---|
| Inventory | Purchased versus discovered device/path map | Devices missing or topology unexplained |
| Local readiness | DCGM results, exit codes, skipped-test list | Required checks fail or were never exercised |
| Local transfers | Chosen nvbandwidth matrices with test method | Unexplained outliers or mismatched measurement types |
| Two-node fabric | Placement, correctness results, transport evidence | Wrong transport or unverified participation |
| Scale-out | Per-size results across topology boundaries | Agreed floors fail or coverage is incomplete |
| Contention | Matched isolated and overlapping-workload records | Application objectives fail without an accepted limitation |
| Recovery | Approved drill logs and restoration checks | Recovery differs from the agreed operating model |
| Handover | Reproducible bundle, owners and exception register | Exceptions lack ownership, expiry or remediation |
For each row, name the owner and reviewer. Distinguish fail, pass, waived and not tested. A waiver must state its operational impact and expiry; it should never be converted into a green cell simply to finish the spreadsheet. Keep performance exceptions separate from correctness or safety exceptions.
10. Alternatives and practical takeaways
For a small lab, use a reduced matrix with explicit coverage gaps rather than pretending to deliver production certification. For a production purchase, require a witnessed supplier baseline plus independent reproduction. For a shared cluster, perform intrusive commissioning before general admission and use a smaller approved regression subset thereafter.
The key buying decision is not “which tool gives the biggest bandwidth number?” It is which evidence demonstrates that the delivered system matches the topology, workload and recovery model being purchased? DCGM, nvbandwidth and NCCL tests contribute different pieces; none substitutes for the full acceptance contract.
Continue with the earlier NCCL Slow AllReduce troubleshooting matrix when a gate fails. Use the checkpoint storage sizing guide for overlapping storage tests and the rack power/cooling worksheet for facilities readiness. The AI Infrastructure hub and Data Center hub collect the wider design series.
Summary: freeze the build, verify local health, prove placement and transport, expand across real topology boundaries, test realistic overlap, and rehearse approved recovery. Accept the evidence bundle—not a screenshot of one fast run.
Post a Comment