Most AI-cluster articles describe products. Engineers usually need a different answer: how many ports, switches, links and watts are implied by the GPU count? This page starts that answer with a practical calculator and a repeatable sizing method for early design conversations.
The calculator below is intentionally conservative: it performs visible arithmetic for GPU count, network links and switch-port demand. It does not pretend to be a validated reference architecture, a quote, or a benchmark. Use it to catch impossible assumptions before a vendor design review.
AI GPU cluster network calculator
Use this quick planner to estimate GPU count, host-facing fabric ports, theoretical per-node injection, rough leaf count and first-pass switch-port demand. It is capacity arithmetic, not a validated NVIDIA reference design.
Why this is more useful than another generic AI fabric overview
A buyer, network engineer or data-center planner rarely gets stuck on the definition of InfiniBand. They get stuck when a proposal says “256 GPUs with non-blocking 400G fabric” but nobody has shown the port count, failure domain, storage network or power envelope. A simple calculator makes the discussion auditable.
For example, an eight-GPU server with eight 400Gb/s cluster links injects a theoretical 3.2Tb/s, or 400GB/s, into the compute fabric in one direction. Thirty-two such servers create 256 GPUs and 256 host-facing fabric ports before any leaf-to-spine uplinks, storage network, in-band management or BMC network is counted. NVIDIA’s DGX H100/H200 and DGX B200 documentation both show eight-GPU systems with cluster interfaces capable of up to 400Gb/s InfiniBand, while also separating storage and management interfaces from the cluster network.[1][2]
Quick sizing table: 8-GPU nodes, 400G links, 1:1 fabric
| Nodes | GPUs | Speed | Host ports | Leaf switches | Spines | Total switch-port ends | Per-node injection |
|---|---|---|---|---|---|---|---|
| 4 | 32 | 400G | 32 | 1 | 1 | 64 | 3.2 Tb/s |
| 8 | 64 | 400G | 64 | 2 | 1 | 128 | 3.2 Tb/s |
| 16 | 128 | 400G | 128 | 4 | 2 | 256 | 3.2 Tb/s |
| 32 | 256 | 400G | 256 | 8 | 4 | 512 | 3.2 Tb/s |
| 64 | 512 | 400G | 512 | 16 | 8 | 1024 | 3.2 Tb/s |
The table assumes 64 logical ports per leaf switch, eight 400G links per node and simple 1:1 leaf/spine accounting. It is a first-pass estimator. A real design must account for rail optimization, physical OSFP breakout/cabling, switch SKU, subnet-manager design, route policy, maintenance domains and growth ports.
What the calculator deliberately does not hide
- Logical port count is not cable count. OSFP cages, twin-port adapters, breakout cables and optics can change the physical bill of materials.
- 400G/800G is line rate, not application throughput. NCCL, rank placement, PCIe/NVLink topology, message size and congestion determine observed performance.
- Compute fabric is not the whole network. Storage/checkpoint traffic and management access need separate capacity and failure analysis.
- Power can dominate the design. NVIDIA lists 10.2kW max for DGX H100/H200 and 14.3kW max for DGX B200, so 32 servers are a facility conversation, not only a switch conversation.[1][2]
InfiniBand NDR vs XDR: choose the assumption explicitly
NDR 400G and XDR 800G should be separate design inputs, not marketing adjectives. If you switch the calculator from 400 to 800Gb/s, the per-node injection doubles, but the operational questions do not disappear: host adapter support, switch-port count, optics reach, cabling plan, congestion behavior, telemetry, software versions and acceptance tests still need validation. NVIDIA’s DGX SuperPOD B200 reference architecture separates compute fabric, storage fabric, in-band management and out-of-band management as design components, which is the right mindset for any serious AI cluster plan.[3]
How to use this in a real project
- Start with workload: training, fine-tuning, inference, checkpoint size and recovery objective.
- Select node type and count: GPUs per node, memory per GPU, local NVLink/NVSwitch boundary.
- Set fabric assumption: 100G/200G/400G/800G, links per node and oversubscription target.
- Calculate ports, then add storage and management networks separately.
- Draw the rail or leaf/spine topology and mark failure domains.
- Turn arithmetic into acceptance tests: physical link health, fabric counters, NCCL tests, application job, checkpoint/restore and failure recovery.
Validation checklist
- Every GPU has an intended NIC/rail path documented.
- Every logical fabric link maps to a switch port, module/cage and cable ID.
- Subnet-manager or fabric-manager ownership is clear.
- NCCL troubleshooting starts with topology, GPU-to-NIC path and fabric health, not random tuning variables. NVIDIA’s NCCL troubleshooting guide explicitly covers GPU-to-GPU, GPU-to-NIC, IOMMU/ACS, topology and InfiniBand/RoCE diagnostics.[4]
- Storage throughput is calculated from checkpoint and restore objectives.
- Power, cooling, rack loading and maintenance states are approved before ordering accelerators.
Next step
This calculator is the first version of a more practical AI infrastructure toolset for Network freak. The next upgrades should add rail-optimized fabrics, switch SKU templates, storage/checkpoint sizing and a GPU rack power estimator. For related background, see the AI Infrastructure and Automation hub and the Data Center Networking hub.
Post a Comment