How AI Data Centers Are Wired: Scale-Up, Scale-Out and Scale-Across Networks Explained
Deep dive, written 20 November 2026. It is built from what Meta, Microsoft, Google, NVIDIA and the standards bodies have published, and every number links to its source at the bottom. Where a figure is our own arithmetic, the text says so.
A cloud data center connects many small, independent workloads. An AI data center does the opposite: it connects tens or hundreds of thousands of GPUs that must behave like one computer. Microsoft describes its Fairwater site as "one massive AI supercomputer using a single flat networking interconnecting hundreds of thousands" of GPUs.[6] Meta's Prometheus spans several buildings while "functioning as a single machine".[4] That changes almost every network design rule we learned in the leaf-spine era: how many networks you build, which protocols carry the traffic, how you balance load, how much buffer you need, and even how many floors the building has.
- Three layers, not one. Every large AI site has a scale-up domain (NVLink inside a 72-GPU rack), a scale-out backend (RoCE or InfiniBand between racks) and, increasingly, a scale-across layer that joins buildings and regions.[13][5]
- The backend is separate and non-blocking. Training traffic gets its own fabric at 400-800G per GPU, usually rail-optimized; data ingest, storage, users and management ride other networks.[1][11][7]
- Ethernet is winning the scale-out fabric. Meta, Microsoft and Google run Ethernet-based backends; NVIDIA's reference design still uses InfiniBand. The new Ultra Ethernet transport adds packet spraying and modern congestion control.[4][7][8][11][17]
- ECMP alone is not enough. AI traffic is a few line-rate flows that start together; operators moved to adaptive routing, per-packet spraying and multi-plane fabrics.[1][5]
- Distance is now a design input. Light needs about 5 µs per km of fiber, so links between sites need deep buffers or distance-aware congestion control, plus 800G-ZR coherent optics.[13][4][5]
- Power and cooling shape the network. GPU racks draw 120-170 kW and need liquid cooling; Microsoft built two-story halls just to shorten cables.[12][7][4]
Why AI traffic is different · The separate networks · Scale-up · Scale-out fabric · Protocols · Load balancing · Congestion control · Frontend, storage, OOB · Connecting data centers · Physical infrastructure · Reference designs · Design checklist · FAQ
Why AI traffic is different
Large models are trained in steps. In each step every GPU computes on its share of the work and then exchanges results with the others through collective operations such as AllReduce, AllGather and All-to-All. The next step cannot start until the slowest exchange finishes. In practice, the network is judged by the tail of its latency and by how evenly it spreads load, not by average throughput.
Meta lists the properties that make this traffic hard for a classic data center fabric:[1]
- Low entropy: "the number and the diversity of flows for AI workloads are much smaller", and the patterns repeat.
- Burstiness: flows switch "on and off" at millisecond granularity.
- Elephant flows: "the intensity of each flow could reach up to the line rate of NICs".
Google describes the same thing as "synchronized bursts, i.e., intense, coordinated, millisecond-level traffic spikes", with strict bounds on tail latency.[8] Reliability is tied to performance too. One faulty host can stall a whole job, so Meta tracks training goodput. On Prometheus it reached more than 85% for pre-training and a mean time between interruptions of about 47,000 GPU hours, later raised to over 54,000.[4]
One cluster, several separate networks
The first design rule is separation. Meta built "a dedicated backend network specifically for distributed training", so it can "evolve, operate, and scale independently from the rest of the data center network". The frontend handles "data ingestion, checkpointing, and logging".[1] Google splits its AI network into "a scale-up domain for intra-pod connectivity, a dedicated east-west scale-out accelerator fabric, [and] the Jupiter frontend network for north-south compute and storage access".[8]
NVIDIA's DGX SuperPOD reference architecture is the most explicit. It uses five logical networks (NVLink, compute fabric, storage, in-band management, out-of-band management) carried over four physical fabrics: the multi-node NVLink fabric, the compute InfiniBand fabric, a combined storage and in-band Ethernet fabric, and the out-of-band network.[11] On a GB200 compute tray this shows up as separate ports: four ConnectX-7 400G NICs for the cluster network (ConnectX-8 at 800G on GB300), two BlueField-3 DPUs for storage and management, and 1GbE BMC ports for out-of-band access.[12]
Scale-up: the rack becomes one accelerator
Inside a rack, GPUs talk over a memory-semantic interconnect rather than a network stack. In an NVL72 rack there are 18 compute trays with four GPUs each and nine NVLink switch trays, each with two NVSwitch chips. Each GPU has 18 NVLink 5 links, one to each switch chip, for 1.8 TB/s of bandwidth and full-mesh connectivity between all 72 GPUs.[11] The links run over a passive copper cable cartridge backplane, and the rack draws about 120 kW.[12] Microsoft notes that this makes the rack behave like "a single, giant accelerator" with 14 TB of pooled memory.[6]
A bigger scale-up domain also means a bigger blast radius. Meta points out that going from 8-GPU servers to NVL72 increased "the fault blast radius by 9X". Its answer is to run a 64-GPU slice per rack and keep eight GPUs as in-rack spares. For a 32,768-GPU job it provisions 523 racks, about 13% more GPUs than the job uses. Signal integrity is the other problem: one NVL72 domain has 5,184 links at 200 Gbps, and Meta screens more than 600,000 NVLink paths before production.[4]
NVLink is proprietary, and there are now two open alternatives:
- UALink 1.0 (April 2025) defines a memory-semantic (read, write, atomic) scale-up link at 200G per lane "for up to 1,024 accelerators within an AI computing pod".[18]
- Ethernet for scale-up. Broadcom's Tomahawk 6 supports scale-up clusters of 512 XPUs.[19] Meta's Tahan switch, built on a 51.2T Tomahawk 5, connects 16 MTIA accelerators and reuses "the same lossless RoCE transport (routed IPv6, PFC, AI-tuned QoS)" as its scale-out network. The OCP ESUN group published an ESUN 1.0 requirements specification in 2026.[5]
Scale-out: the backend fabric
The scale-out fabric connects scale-up domains into a cluster. Its defining properties are bandwidth per GPU and non-blocking behaviour. Meta's backend is "a non-blocking architecture, providing high bandwidth, low latency, and lossless transport between any two GPUs".[1] Microsoft uses "a full fat tree non-blocking architecture" at 800 Gbps per GPU.[6][7] Bandwidth per GPU keeps growing. Meta paired each GB200 GPU with two 400G NICs to get 800G of scale-out bandwidth early, because the other dimensions of the platform had grown while 400G per GPU had not.[4]
Rail-optimized topology
Most backends are rail-optimized. In a server or tray with K GPUs there are K rails, and a rail is made of "GPUs with the same local rank that belong to different" servers. All GPUs of a rail sit under the same rail switch, because "an optimal DNN parallelization strategy concentrates its NIC domain traffic between GPUs with the same local rank".[21] NVIDIA's GB200 design is "rail-aligned": traffic on the same rail "is always one hop away" inside a scalable unit (SU), while traffic between racks or rails "traverses the spine layer".[11] An SU is eight NVL72 racks, or 576 GPUs, and the InfiniBand fat tree scales to 16 SUs (9,216 GPUs) with spine-leaf groups and a core layer.[11]
How big can one fabric be?
Switch radix sets the limit. A two-tier Clos of k-port switches supports k²/2 hosts at full bandwidth, and three tiers support k³/4. With 51.2T chips (64 × 800G) that is 2,048 GPUs in two tiers. With 102.4T chips such as Tomahawk 6 it is 8,192 at 800G, or about 131,000 if the same chip is split into 512 ports of 200G, which matches Broadcom's claim of "100,000+ XPUs in a two-tier scale-out network at 200 Gbps/link".[19]
That is why multiplane fabrics are spreading. In Meta's Flattened Planar Fabric, "each GPU connects via eight independent 100G links to eight different switches (one per network plane)". This took Meta from a three-tier to a two-tier Clos, which "reduces network hops, latency, and infrastructure cost", and the NIC sprays packets across all eight planes.[5] Google's Virgo network for TPUs is likewise "a flat, two-layer non-blocking topology" with a "multi-planar design" and links 134,000 TPU 8t chips with up to 47 Pb/s of bisection bandwidth.[8]
InfiniBand or Ethernet?
Both work at scale. In 2024 Meta built two 24,576-GPU clusters, one with a RoCE fabric (Arista 7800 with Wedge400 and Minipack2 rack switches) and one with NVIDIA Quantum-2 InfiniBand, both at 400 Gbps per endpoint, and ran large GenAI training on both "without any network bottlenecks".[2] The large new builds have moved to Ethernet. Prometheus uses Ethernet building fabrics of up to 20,736 GPUs each, first a Disaggregated Scheduled Fabric (DSF) and then a Non-Scheduled Fabric (NSF) on shallow-buffer 51.2T switches.[4] Microsoft's Fairwater backend is "a two-tier, ethernet-based backend network" on SONiC, chosen in part to "avoid vendor lock-in".[7] (Microsoft's September 2025 post said its pods use "both InfiniBand and Ethernet fabrics".[6]) NVIDIA's DGX SuperPOD reference design still uses InfiniBand for the compute fabric.[11] For a deeper look at the InfiniBand option, see our NDR vs XDR rail design guide.
Protocols: how bytes move between GPUs
All scale-out options are built on RDMA: the NIC reads and writes remote memory directly, without the remote CPU. Collective libraries such as NCCL run on top.
- InfiniBand has its own link, network and transport layers and lossless links. NVIDIA adds SHARP, which offloads collectives into the switches and reduces "the amount of data traversing the network as aggregation nodes are reached".[15]
- RoCEv2 carries the InfiniBand transport over UDP/IP (UDP port 4791), so it can be routed over a normal IP fabric.[14][1] It needs a lossless class: NVIDIA states that "RoCEv2 requires flow control for lossless Ethernet" and that "RoCE relies on ECN and PFC to operate".[14]
- Ultra Ethernet Transport (UET), part of the UEC 1.0 specification published in June 2025, makes RDMA "a native Ethernet application" and implements a native transport for the libfabric 2.0 API.[16][17] Compared with traditional RDMA it offers per-packet multipathing (spraying), native out-of-order delivery, per-packet loss recovery instead of go-back-N, fast "ephemeral" connections, packet trimming, sender-based (NSCC) and receiver-based (RCCC) congestion control, and optional group-based encryption. It is designed for more than a million endpoints.[17]
The underlay of an Ethernet backend is usually plain routed IP. eBGP is the common choice in large data centers because of its "operational simplicity and network stability" (RFC 7938).[22] Meta even uses eBGP with link-bandwidth attributes for unequal-cost load balancing between buildings, as described below.[3] If you build the frontend or a small GPU pod yourself, our BGP unnumbered leaf-spine guide shows the pattern.
Load balancing: why ECMP is not enough
Classic ECMP hashes each flow's five-tuple onto one path. With thousands of small flows that averages out. With a handful of line-rate flows it does not. Meta's experience is the best public record of this:[1]
- Plain ECMP "rendered poor performance for the training workload due to the low flow entropy".
- Path pinning by destination worked only when racks were fully used by one job. With fragmented placement it "degraded the training performance up to more than 30%".
- As a stop-gap Meta doubled uplink bandwidth (1:2 under-subscription), which it called "an expensive solution".
- Enhanced ECMP, which also hashes on the RoCE destination queue pair, together with queue-pair scaling in the collective library, gave "performance improvement of up to 40% for the AllReduce collective".
The current answer is to stop thinking in flows. Switches with adaptive routing or dynamic load balancing pick the least-loaded path per packet, and NICs that can reorder packets make that safe. Prometheus' NSF fabric uses Minipack3 switches with Dynamic Load Balancing and NVIDIA Spectrum-4 based switches with NVIDIA Adaptive Routing on ConnectX NICs.[4] Meta's FPF moves load balancing into the NIC, which "sprays every packet across all eight planes in hardware".[5] Microsoft lists "packet spray" among the core components of the Fairwater network.[7] UET builds spraying into the transport itself.[17]
Congestion control and lossless Ethernet
RoCE traditionally combines PFC, which pauses one priority class hop by hop, with ECN-based end-to-end rate control (DCQCN). In NVIDIA's Cumulus defaults RoCE uses switch priority 3, PFC on that priority in lossless mode, and ECN marking.[14] Real deployments are less tidy. When Meta moved to 400G, DCQCN with doubled ECN thresholds degraded performance, and firmware changes introduced bugs. Meta proceeded with PFC only and "over a year of experience with just PFC" showed "stable performance". It limits in-flight data through receiver-driven admission instead: the NCCL sender only posts an RDMA write after a clear-to-send from the receiver, and CTS packets get high-priority queues in the switches.[1]
Newer tools target the same goal of reacting in microseconds. Packet trimming lets a switch notify "both receiver and sender of packet loss and congestion, allowing rapid mitigation and recovery".[17] Broadcom's Tomahawk 6 combines telemetry, dynamic congestion control, rapid failure detection and trimming.[19] Microsoft relies on "packet trimming, packet spray and high-frequency telemetry".[7] Google uses "sub-millisecond telemetry to identify elusive network micro-bursts that are usually missed by conventional 30-second monitoring intervals".[8] If you run RoCE today, our PFC buffer sizing worksheet and RoCEv2 troubleshooting checklist cover the switch side.
Frontend, storage and management networks
The backend gets the headlines, but the other networks decide whether GPUs stay busy. Storage carries datasets in and checkpoints out, and thousands of GPUs save checkpoints at the same moment. Meta built its checkpointing to "run in hundreds of milliseconds" across thousands of ranks.[2] Microsoft built "an entirely separate datacenter infrastructure" to store and process data for Fairwater; its storage systems are "five football fields in length".[6]
In NVIDIA's GB200 SuperPOD, storage and in-band management share one Ethernet fabric built from SN5600 switches, with a super-spine layer for multi-SU builds. The storage network runs as a dedicated VXLAN segment with VTEPs on the leaf switches, uses RoCE with congestion control and adaptive routing, and gets 16 × 800 Gbps non-blocking bandwidth to the storage appliances per SU, with a 5:3 blocking factor on the GPU side. The hand-off to the customer network is eBGP, with at least 2 × 100GbE recommended, and the cluster now also needs a link to the building management system (BMS).[11] The out-of-band network connects every BMC, PDU and switch management port and also carries InfiniBand and NVLink fabric management.[11] Our posts on AI storage network design, checkpoint storage sizing and out-of-band network design go deeper.
Scale-across: connecting data centers into one cluster
Power is the limit on how big one building can get. Google puts it simply: "it is far more difficult to move electrons (electrical power) than it is to move photons (data over fiber)". It places data centers near energy and uses the network "to distribute AI workloads across campuses".[8] NVIDIA calls this third dimension scale-across, which applies to links "over 500 meters", from buildings on a campus to sites in different states.[13]
Meta Prometheus is the most detailed public example:[3][4][5]
- The backend aggregation (BAG) layer is an Ethernet super-spine that connects building fabrics "spanning 2 km to over 100 km". It is built from Arista 7800-series chassis with deep-buffered Broadcom Jericho3 line cards, with up to 432 × 800G ports, and carries 16-48 Pbps between a pair of regions.
- Oversubscription from building fabric to BAG is "around 4.5:1". Training jobs are placed so that heavy traffic stays inside a building.
- Routing is eBGP with link-bandwidth attributes, which gives unequal-cost multipath (UCMP) so that traffic follows real capacity. Links between BAGs are encrypted with MACsec.
- Optics range from "2×400G FR4 LITE short-reach, low-cost pluggables inside the buildings to 800G-ZR coherent optics for links beyond ~10 km".
- Regional backbones use SRv6 on FBOSS and Cisco Silicon One based switches.
Microsoft built a dedicated "AI WAN optical network" and added "over 120,000 new fiber miles across the US" in a year. The Atlanta and Wisconsin Fairwater sites form what Microsoft calls an AI superfactory.[7] Google's Virgo "can expand across multiple data centers", and its global network grew 10x in WAN traffic between 2020 and 2025.[8]
Distance changes the physics. Light in fiber takes about 5 µs per km.[13] At 100 km a round trip takes about 1 ms, and an 800G link holds around 100 MB in flight (our arithmetic). A lossless link must buffer everything already in flight when it sends a pause, and Meta notes that "lossless Ethernet requires buffer headroom proportional to cable length". Its FBOSS now measures real cable length and sizes headroom from that measurement.[5] You can check the delay for your own distances with our network latency calculator.
The industry has two answers. Meta uses deep-buffer switches in the BAG role.[3] NVIDIA argues the opposite: deep buffers add latency and jitter that synchronous training cannot tolerate. Its Spectrum-XGS uses "distance-aware" congestion control and adaptive routing instead and claims up to 1.9x higher NCCL all-reduce bandwidth at 10 km than "off-the-shelf Ethernet".[13] Both sides agree that software matters as much. Meta schedules so that "latency-sensitive and bandwidth-sensitive operations concentrate within a replica" while more tolerant traffic crosses buildings, and it warns that collective libraries that ignore distance "can fall off a performance cliff".[4][5]
For the optics, the OIF 800ZR implementation agreement (October 2024) defines interoperable 800G coherent pluggables for "single-span, amplified 80-120 km" DWDM links, aimed at data center interconnect.[20] Use our optical power budget calculator for a first link-budget check.
Physical infrastructure: what you build before the first packet
- Power density. A GB200 NVL72 rack draws about 120 kW, fed through 50 V DC busbars from eight power shelves.[12] Fairwater is designed for "~140kW per rack, 1,360 kW per row".[7] Meta's Vera Rubin NVL72 racks will exceed 170 kW.[4]
- Liquid cooling. Compute trays are cooled with cold plates and manifolds.[12] Fairwater and Meta's AI buildings use closed-loop liquid cooling,[7][4] and the network is following: Meta's IcePack is a liquid-cooled rack for its 102.4T IceCube switches (128 × 800G).[5]
- Cable length as a design input. Because "cable run lengths can meaningfully impact cluster performance", Microsoft built Fairwater as a two-story building that places racks "in three dimensions to minimize cable lengths".[7]
- Optics and fiber at huge scale. Prometheus involves "millions of miles of fiber strand, tens of thousands of switches, and millions of optics".[4] 102.4T switch chips come with 200G SerDes, long-reach passive copper and a co-packaged optics option.[19]
- Power swings. Synchronous jobs make the whole cluster's load jump by 30-40%, and starting or stopping a large job can move "tens of megawatts in just seconds". Meta smooths this with GPU power capping, rack batteries, capacitor trays and site batteries.[4] Microsoft uses similar software, hardware and energy-storage measures.[7]
- The building is part of the network. NVIDIA's reference design now connects the cluster to the building management system,[11] and Google's OCS layer became "part of the building infrastructure", independent of data rate.[9]
For a smaller build, our GPU rack power and cooling sizing guide covers the facility side for a 256-GPU cluster.
Reference designs side by side
One more design deserves a mention because it goes against the trend: Google's Jupiter data center network replaced the spine layer with optical circuit switches (OCS). MEMS mirrors map fibers port to port, so the network topology can be reconfigured in software. Google reports 40% less power, 30% lower cost and 50x less downtime than the best alternatives it knew of.[9] TPU v4 pods use the same idea: 4,096 chips connected by OCS, with optics making up less than 5% of system cost and power.[10]
Designing an AI fabric: a practical checklist
Most readers will not build a gigawatt cluster, but the same rules apply to a 256-GPU pod. This list is our own practice, drawn from the designs above:
- Separate the backend. GPU-to-GPU traffic gets its own fabric. Storage, users, scheduling and management go elsewhere, and out-of-band stays independent.
- Size bandwidth per GPU first, then pick the topology. Rail-optimized, non-blocking, with as few tiers as the switch radix allows. Our AI GPU cluster network calculator does the port and switch math for 8-512 GPUs.
- Choose the transport and stick to one lossless design. InfiniBand, or RoCEv2 with a single lossless priority, consistent PFC and ECN thresholds on every hop, and NICs that support adaptive routing or spraying.
- Plan for collisions, not averages. Turn on adaptive routing or DLB, check queue-pair counts in NCCL, and test with the real collective sizes.
- Measure what matters. Per-queue drops, PFC pause counters, ECN marks and sub-second telemetry, plus NCCL bus bandwidth per job. Our acceptance test plan and slow AllReduce troubleshooting matrix show how.
- Keep distance in mind. Anything beyond one hall needs a buffer and congestion-control plan, and job placement that keeps heavy collectives local.
- Design for failure. Spare capacity in every scale-up domain, fast drain of failed links and checkpoint storage sized for burst writes.
FAQ
What is the difference between scale-up, scale-out and scale-across?
Scale-up links GPUs inside a rack or pod into one memory domain (NVLink, UALink, Ethernet scale-up). Scale-out connects many such domains in a data center (InfiniBand, RoCE, Ultra Ethernet). Scale-across connects data centers that are more than about 500 m apart into one cluster.[13]
Do AI data centers still use BGP and EVPN-VXLAN?
Yes. eBGP is the usual routing protocol for large IP fabrics[22] and Meta uses it with link-bandwidth attributes between buildings.[3] VXLAN segments appear in the storage and in-band network of NVIDIA's reference design, with eBGP to the customer edge.[11] See our BGP EVPN-VXLAN checklist for GPU pods.
Is InfiniBand dead for AI?
No. NVIDIA's DGX SuperPOD still specifies InfiniBand for the compute fabric, and Meta ran an InfiniBand cluster alongside its RoCE cluster.[11][2] But the biggest published builds from Meta, Microsoft and Google use Ethernet-based fabrics.[4][7][8]
How far apart can two data centers be and still train one model?
Meta's BAG layer already spans more than 100 km.[4] The limit is not the fibre itself but latency and buffers: about 5 µs per km, which must be absorbed by switch buffers or by distance-aware congestion control, plus software that keeps the heaviest traffic inside each site.[13][5]
Try it: Network Latency Calculator (fiber delay per km) and Optical Power Budget Calculator - free, run in your browser. All tools: Free Networking Tools.
Sources (all read on 20 November 2026)
[1] Meta Engineering, "RoCE networks for distributed AI training at scale," 5 Aug 2024 (summary of the SIGCOMM 2024 paper "RDMA over Ethernet for Distributed AI Training at Meta Scale").
[2] Meta Engineering, "Building Meta's GenAI Infrastructure," 12 Mar 2024.
[3] Meta Engineering, "Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters," 9 Feb 2026.
[4] Meta Engineering, "Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster," 19 Oct 2026.
[5] Meta Engineering, "OCP Summit 2026: Powering AI with Open Innovations in Scale-Up, Scale-Out, and Scale-Across Networks," 12 Oct 2026.
[6] Microsoft, "Inside the world's most powerful AI datacenter," 18 Sep 2025.
[7] Microsoft, "Infinite scale: The architecture behind the Azure AI superfactory," 12 Nov 2025.
[8] Google Cloud, "How we evolved Google's global and data center networks for the AI era," 27 May 2026.
[9] Google Cloud, "Jupiter evolving: Reflecting on Google's data center network transformation," 24 Aug 2022.
[10] Google Cloud, "Google's Cloud TPU v4 provides exaFLOPS-scale ML with industry-leading efficiency," 5 Apr 2023.
[11] NVIDIA, "DGX SuperPOD GB200 Reference Architecture: Network Fabrics."
[12] NVIDIA, "DGX GB Rack Scale Systems User Guide: Hardware."
[13] NVIDIA Technical Blog, "How to Connect Distributed Data Centers Into Large AI Factories with Scale-Across Networking," 9 Sep 2025.
[14] NVIDIA, "Cumulus Linux 5.13: RDMA over Converged Ethernet (RoCE)."
[15] NVIDIA, "Scalable Hierarchical Aggregation and Reduction Protocol (SHARP): Introduction."
[16] Ultra Ethernet Consortium, "UEC Launches Specification 1.0," 11 Jun 2025.
[17] Arista, "Demystifying Ultra Ethernet."
[18] UALink Consortium, "UALink Consortium Releases the Ultra Accelerator Link 200G 1.0 Specification," 8 Apr 2025.
[19] Broadcom, "Broadcom Ships Tomahawk 6: World's First 102.4 Tbps Switch," 3 Jun 2025.
[20] OIF, "OIF Releases 800ZR Coherent Interface Implementation Agreement," 30 Oct 2024.
[21] Wang, Ghobadi et al., "Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters," arXiv 2307.12169.
[22] IETF, "RFC 7938: Use of BGP for Routing in Large-Scale Data Centers," 2016.
Diagrams: Network freak, based on the sources above. Photos: Wikimedia Commons under the licenses named in each caption; resized.
Related reading
- NVIDIA InfiniBand for AI Clusters: NDR vs XDR, Rail Design and Acceptance Tests
- AI GPU Cluster Network Calculator: Ports, Bandwidth and Switch Sizing for 8–512 GPUs
- RoCE PFC Buffer Sizing Guide: 400G Headroom Worksheet and ECN Checklist
- NCCL Slow AllReduce Troubleshooting: GPU, RDMA and Rail Test Matrix
- AI Cluster Storage Network Design: Stop Backups from Breaking GPU Jobs
- AI Cluster Networking: Practical BGP EVPN/VXLAN Checklist for Small GPU Pods
Comments
Post a Comment