AI cluster networking often fails in ordinary places: an underlay route is missing, an EVPN route target is wrong, a VTEP is silent, or storage traffic accidentally shares a failure domain with GPU traffic. The hardware may be expensive, but the most reliable design pattern is still a boring, testable leaf-spine fabric with clear separation between underlay, overlay and tenant policy.
This practical checklist is written for small and medium AI labs, private GPU pods and data-center teams that want a repeatable BGP EVPN/VXLAN design without turning every troubleshooting session into a vendor-specific archaeology project.
Problem: AI clusters are sensitive to small network mistakes
A web application can usually hide a little jitter behind retries and queues. Distributed training is less forgiving. When the cluster depends on synchronized GPU workers, fast checkpoint writes and predictable east-west flows, one weak rack uplink or one asymmetric path can look like an application bug.
The common symptoms are practical, not theoretical:
- Training jobs slow down after a node or rack move.
- Storage looks reachable but checkpoint writes are inconsistent.
- Only one tenant, VLAN or VRF fails after a maintenance window.
- Operators see clean interfaces but missing MAC/IP routes in the overlay.
- Firewall or backup paths quietly become bottlenecks for traffic that should stay local.
If you are new to the blog, the Start Here networking topics page is a good entry point. For related design notes, see the AI Infrastructure and Data Center hubs.
Design: use a boring underlay and a policy-driven overlay
The cleanest mental model is to separate transport from tenancy:
- Underlay: point-to-point routed links, loopbacks and ECMP between leaf and spine switches.
- Overlay: EVPN control plane carrying MAC/IP and IP prefix reachability.
- Tenants: VRFs, VNIs, route targets and explicit policy between GPU, storage, backup and management domains.
Underlay recommendations
Use eBGP or iBGP consistently. For smaller fabrics, eBGP between leaf and spine is easy to reason about because every adjacency is explicit. Advertise loopbacks, keep timers conservative unless you have tested convergence, and avoid mixing too many design patterns in one pod.
interface Ethernet1/1
description to-spine-1
no switchport
ip address 172.31.0.1/31
router bgp 65011
router-id 10.255.0.11
neighbor 172.31.0.0 remote-as 65000
address-family ipv4 unicast
network 10.255.0.11/32The addresses above are documentation-only examples. Replace them with your own addressing plan and keep real production addresses out of public notes and tickets.
Overlay recommendations
EVPN/VXLAN is useful because it gives the fabric a control plane. Do not treat VXLAN as “just tunneling”. The real operational value is in route targets, route types and predictable VTEP behavior.
For a small AI pod, start with a few clear tenants:
- GPU / training tenant: high-throughput east-west traffic, strict QoS validation and minimal unnecessary routing.
- Storage / checkpoint tenant: access to storage systems, backup windows and well-defined firewall policy.
- Management tenant: OOB or logically isolated management, jump host access and MFA.
Implementation workflow
1. Build the underlay first
Before configuring EVPN, prove that every leaf can reach every VTEP loopback through the spine layer. If this is not true, overlay troubleshooting becomes noise.
# Minimum checks
show ip bgp summary
show ip route 10.255.0.0/24
ping 10.255.0.12 source 10.255.0.11
traceroute 10.255.0.12 source 10.255.0.112. Add EVPN and route targets deliberately
Most “mysterious” EVPN outages are simple import/export mistakes. Keep route-target naming systematic and document which VNI belongs to which tenant.
vrf context AI-GPU
vni 10110
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vlan 110
vn-segment 10110Auto-derived route targets can be fine in a controlled fabric, but be careful when you interconnect pods, stretch tenants or integrate a firewall. At those boundaries, explicit route targets are often easier to audit.
3. Keep backup and management paths boring
Backup and ransomware segmentation matter even in AI labs. A backup network that can mount every dataset and every checkpoint should not be casually reachable from the same tenant used for experiments. Put it in a separate VRF/VNI, restrict lateral movement and log policy hits.
If you need external references while building your checklist, the Resources page collects useful BGP, networking and AI infrastructure links. For BGP visibility habits, the BGP Table Watch hub is also relevant.
Verification and troubleshooting
Use a fixed order every time. Random command hunting wastes time and often hides the real failure domain.
Physical and L1/L2 checks
- Check CRC, symbol, FEC and flap counters before blaming BGP.
- Confirm breakout and speed settings, especially on dense GPU racks.
- Look for pause/PFC counters if the workload uses RoCE-style lossless behavior.
Underlay checks
- Every leaf should learn every VTEP loopback through more than one spine when redundancy exists.
- ECMP next-hops should be balanced and deterministic enough for your traffic profile.
- Route policies should not accidentally suppress loopbacks or infrastructure prefixes.
Overlay checks
- Verify EVPN neighbor state and route counts before testing tenant pings.
- Check type-2 MAC/IP routes for host reachability and type-5 routes for routed tenant prefixes.
- Confirm VLAN-to-VNI mapping on the local leaf where the server is attached.
Tenant and policy checks
- Test within a rack, across racks and through the firewall as separate cases.
- Confirm that backup networks are reachable only from expected systems.
- Keep a small “known good” test host or container per tenant for maintenance windows.
Practical takeaways
- Do not start with exotic fabric features. Start with clean routed underlay, stable loopbacks and observable EVPN.
- Document every tenant: VRF, VLAN, VNI, route target, gateway location and firewall path.
- Troubleshoot in order: link, underlay, EVPN, VTEP, tenant policy, then workload.
- Separate GPU, storage, backup and management domains early. It is cheaper than retrofitting segmentation after a security incident.
- Use automation for snapshots: capture BGP state, EVPN routes, interface counters and policy hits before and after every change.
A simple AI cluster fabric is not less professional than a complicated one. It is easier to test, easier to automate and much easier to recover when the next training job exposes a hidden network assumption.
Post a Comment