AI-ready leaf-spine network design sounds like something only hyperscalers need, but the same problems now appear in smaller rooms: one rack with two to eight GPU servers, a storage node, a few 25/100GbE links, and a team asking why training jobs slow down when backups or tenant traffic start moving.
This guide uses a concrete scenario: a small company or lab is building a 4-node GPU cluster. Each server has two 25GbE NICs for general traffic and optionally one 100GbE NIC for GPU/RDMA traffic. The goal is not to copy a hyperscale fabric. The goal is to make sane purchasing and configuration decisions before the first expensive GPU server is cabled.
Problem: the GPU servers are fast, but the network is accidental
A common first build looks like this: all servers connect to one large switch, storage uses the same VLAN as management, and every application team asks for a different exception in the firewall. It works during the first benchmark. It becomes painful when three things happen at the same time:
- GPU nodes exchange east-west traffic during distributed training.
- Users copy datasets from NAS or object storage.
- Management, backup, monitoring and container overlay traffic share the same uplinks.
The symptom is usually vague: variable job duration, TCP retransmits, storage latency spikes, or packet drops on switch queues. Before buying more GPUs, design the fabric so you can isolate, measure and troubleshoot those flows.
Reference topology: small AI rack with room to grow
For a practical first design, use two leaf switches and two spines. If the budget is tight, start with two leaf switches only, but choose hardware and cabling that allows adding spines later. The design below is deliberately generic and uses documentation prefixes only.
+-----------+ +-----------+
| Spine-1 | | Spine-2 |
| 100/400G | | 100/400G |
+-----+-----+ +-----+-----+
| |
100G/400G | | 100G/400G
+-----+--------------------+-----+
| |
+------+-------+ +-------+------+
| Leaf-1 |================| Leaf-2 |
| 25/100G ToR | MLAG/vPC or | 25/100G ToR |
+---+------+---+ EVPN multihoming+---+------+
| | | |
GPU-1 GPU-2 GPU-3 GPU-4
NAS / backup / telemetry collectors connect on dedicated ports/VLANs
Diagram: generic two-leaf/two-spine AI-ready fabric. No customer topology or real addresses are used.
Diagnosis: what traffic classes need different treatment?
Do not begin with commands. Begin with a traffic inventory. A 4-node GPU cluster commonly needs at least these logical networks:
| VLAN / VRF | Purpose | Example subnet | Notes |
|---|---|---|---|
| VLAN 10 / MGMT | BMC, switch management, hypervisor management | 192.0.2.0/24 | Reachable only from admin jump hosts |
| VLAN 20 / USERS | SSH, notebooks, application access | 198.51.100.0/24 | Firewall policy and MFA matter here |
| VLAN 30 / STORAGE | NFS, SMB, S3 gateway, backup staging | 203.0.113.0/24 | Keep noisy backup jobs predictable |
| VLAN 40 / RDMA | RoCEv2 or high-throughput east-west traffic | 10.40.0.0/24 | Prefer non-routed or tightly controlled at first |
| VLAN 50 / TELEMETRY | SNMP, streaming telemetry, syslog, flow export | 10.50.0.0/24 | Do not mix with user access |
Design options: simple L2, routed leaf-spine, or EVPN/VXLAN?
Option 1: dual ToR with MLAG/vPC. This is the easiest design for one rack. Servers use bonds/LAGs to two switches. Storage and management remain simple. The downside is that expansion across racks becomes vendor-specific and operationally fragile if Layer 2 grows too far.
Option 2: routed leaf-spine with VLANs local to each rack. This is a strong default for teams that can keep most workloads in one rack or one failure domain. It gives predictable ECMP, simple troubleshooting and less broadcast. Use BGP or OSPF underlay, /31 or /30 point-to-point links, and loopbacks for control protocols.
Option 3: EVPN/VXLAN fabric. Choose this when workloads may move between racks, multiple tenants need separation, or you want a consistent model for VRFs and stretched segments. EVPN also gives a clean path to multihoming without relying only on classic MLAG. The trade-off is skill: engineers must understand VTEPs, VNIs, route targets, anycast gateways and BGP EVPN routes.
RoCEv2 basics: be careful with the word “lossless”
RoCEv2 can reduce CPU overhead and latency for some storage and GPU communication patterns, but it is not magic. If you enable it without understanding congestion, you may create a network that fails in less obvious ways. For a small cluster, decide first whether you actually need RDMA. Many early AI labs are better served by clean 25/100GbE TCP, good storage placement and solid monitoring.
If you do need RoCEv2, plan these items explicitly:
- Priority Flow Control (PFC): enable only for the RDMA class, not for every packet.
- ECN/WRED: mark congestion before queues become disasters.
- DSCP/CoS mapping: document the exact marking from server NIC to switch queue.
- Buffer visibility: make sure your switches expose queue drops, pause frames and ECN counters.
- MTU: use a consistent jumbo MTU where required, and test every path.
Practical configuration checklist
Vendor syntax differs, but the design checklist is stable:
- Create a management VRF or isolated management VLAN. Permit SSH/HTTPS/SNMP only from a jump host subnet.
- Use routed point-to-point links between leaf and spine switches. Keep the underlay boring.
- Run eBGP or OSPF in the underlay. For small fabrics, eBGP with private ASNs per leaf/spine is easy to reason about.
- If using EVPN/VXLAN, create loopbacks for VTEPs, define VNIs per VLAN/VRF, and use an anycast gateway for server VLANs.
- Separate storage and RDMA from user access. A compromised notebook should not have direct access to BMC interfaces or backup repositories.
- Apply QoS only after you can measure queues. Bad QoS is worse than no QoS.
- Store switch configuration in Git and use Ansible or another automation tool for repeatable VLAN, interface and telemetry changes.
Verification and troubleshooting before production
Do not accept “links are green” as a validation result. Run a small acceptance test and save the output.
- Physical layer: check optics/DAC inventory, speed, FEC mode, CRC errors and link flaps on every 25/100GbE port.
- MTU: test with
ping -M do -s 8972 <peer>on Linux for jumbo paths. - Throughput: run
iperf3 -P 8between GPU nodes and storage. Compare same-leaf and leaf-to-leaf results. - Loss and latency: run continuous ping while an iperf3 job and storage copy are active. Watch for spikes, not only averages.
- EVPN checks: verify BGP EVPN sessions, MAC/IP route learning, VNI state and anycast gateway consistency.
- RoCE checks: monitor PFC pause frames, ECN marks, queue drops and NIC counters during real workload tests.
- Security matrix: prove that VLAN 20 users cannot reach VLAN 10 BMC addresses, and that backup/storage access is limited to required hosts.
- Automation dry-run: run Ansible in check mode before pushing interface or VLAN changes.
Neutral shopping list / bill of materials
Prices vary by region and vendor, but these tiers help frame the decision. Treat them as planning ranges, not affiliate recommendations.
| Tier | Approx. budget | What to buy | Best fit |
|---|---|---|---|
| Starter | €3,000–€8,000 / $3,000–$8,000 | Two used/refurb 25GbE switches, DAC cables, 25GbE NICs, basic telemetry VM | One rack, TCP workloads, learning EVPN later |
| Balanced | €12,000–€35,000 / $12,000–$35,000 | Two 25/100GbE leaf switches, 100GbE uplink optics/DACs, support, automation host, monitoring | 4–8 GPU servers, storage separation, first production jobs |
| Expandable | €40,000+ / $40,000+ | Two spines, multiple leaf pairs, EVPN/VXLAN licenses/support if required, 100/400G optics, telemetry platform | Multiple racks, tenant separation, RDMA or strict SLA requirements |
Summary: buy observability and operational simplicity, not only bandwidth
The best first AI fabric is not necessarily the fastest one on a slide. It is the one your team can operate at 02:00 when a training job slows down. Start with clear VLANs/VRFs, routed leaf-spine principles, measurable telemetry and repeatable configuration. Add EVPN/VXLAN when the operational benefits justify the control-plane complexity. Add RoCEv2 only when the workload and engineers are ready for congestion-control details.
If you are building a small GPU cluster, write down the traffic matrix before you buy switches. It will prevent more mistakes than any single command.
Related reading
- Start Here: Networking Topics
- Useful Resources
- Data Center articles
- EVPN articles and VXLAN articles
- EVPN VXLAN Troubleshooting: Common Problems and Fixes
- Management Plane Security: Practical Guide for Network Engineers
Have you built a small AI or GPU network with 25/100GbE, EVPN/VXLAN or RoCEv2? Share what worked, what broke, and what you would buy differently next time.
Post a Comment