BGP unnumbered for a leaf-spine underlay is one of those designs that looks small on paper but removes a surprising amount of operational noise. Instead of assigning and tracking an address pair for every point-to-point fabric link, the switches form eBGP sessions over IPv6 link-local addresses and advertise the loopbacks needed by the EVPN/VXLAN overlay.
This article is a practical checklist for engineers building a small data center fabric, AI/GPU cluster network, or lab where the main goal is simple: make the underlay boring, repeatable and easy to troubleshoot before adding VXLAN services.
The problem: numbered underlays do not scale operationally
A classic routed leaf-spine fabric works well, but the address plan can become a spreadsheet exercise. Every leaf-to-spine link needs a unique pair of addresses, every replacement has to preserve the mapping, and every troubleshooting session starts with a question: is the link broken, or did somebody paste the wrong address?
For a two-spine, eight-leaf pod this is still manageable. For repeated pods, labs, staging environments and AI infrastructure where racks are added in waves, it becomes friction. BGP unnumbered reduces that friction by making fabric interfaces nearly identical.
If you are new to the data-center building blocks, start with the Start Here networking topics page and the Data Center hub.
Design: what BGP unnumbered does and does not solve
What changes in the underlay
- Fabric links are routed interfaces, but you do not assign a unique IPv4 address to each link.
- The eBGP neighbor is discovered/reached using the directly connected interface and IPv6 link-local behavior.
- The loopback remains the stable identity for router ID, VTEP source and overlay peering.
- Each leaf-spine link is still its own BGP adjacency, so failures are visible and isolated.
What stays the same
- You still need consistent ASN allocation. A common lab model is one ASN for all spines and a unique ASN per leaf, or private ASNs per device.
- You still need route policy: advertise only loopbacks in the underlay, not every connected subnet.
- You still need MTU discipline before VXLAN encapsulation enters the picture.
- You still need EVPN route-target and VNI design for the overlay.
For AI clusters, this split is useful: the underlay gives predictable ECMP between racks, while the overlay or application design decides where tenant, storage, management and GPU traffic boundaries live. The same principle appears in the AI Infrastructure and Automation hub.
Implementation workflow
1. Build stable loopbacks first
Before touching EVPN, give every switch a stable loopback identity. The underlay should advertise these loopbacks and nothing more. In many designs you will have at least one router ID loopback and one VTEP loopback; in smaller labs they may be the same logical address family object, but keep the purpose clear in documentation.
2. Convert fabric ports into routed interfaces
Every leaf-to-spine connection should be a routed port with LLDP enabled, a consistent MTU and a description that identifies the remote device and port. Avoid using switchport trunks between spines and leaves; VXLAN fabrics need routed underlays, not accidental Layer 2 extension in the core.
3. Apply a repeatable BGP unnumbered template
The exact syntax depends on the network OS, but the intent is similar:
router bgp <leaf-asn>
router-id <loopback-id>
bestpath as-path multipath-relax
neighbor FABRIC peer-group
neighbor FABRIC remote-as external
neighbor FABRIC capability extended-nexthop
interface Ethernet fabric-facing
no switchport
mtu <fabric-mtu>
bgp neighbor interface peer-group FABRIC
Keep the underlay export policy strict. Advertise loopbacks, not server VLANs, not management subnets and not random connected routes. If the platform supports a prefix-list or route-map for loopback-only export, use it from day one.
4. Add EVPN only after underlay ECMP is clean
The overlay should come after the boring checks pass: all fabric BGP sessions are established, every leaf can resolve every remote VTEP loopback, and route counts are symmetrical. Only then enable EVPN address-family, VNI mapping and tenant route targets. If you jump straight into EVPN, you will waste time debugging symptoms instead of causes.
Verification checklist
Use the same checklist every time you turn up a rack:
- LLDP: expected leaf-to-spine neighbors are present on the expected ports.
- Interface state: no error-disabled links, no unexpected speed/duplex negotiation, no MTU mismatch.
- Underlay BGP: every fabric-facing adjacency is Established.
- Loopback routes: every leaf sees every spine and remote leaf loopback through equal-cost paths.
- Overlay BGP: EVPN address-family is activated only between the intended overlay peers.
- Data plane: test east-west traffic with realistic MTU before moving production workloads.
Troubleshooting: solve from the underlay upward
Symptom: EVPN routes are missing
First check whether the VTEP loopbacks are reachable through the underlay. If recursion to the remote VTEP fails, EVPN may look broken even when the EVPN session itself is configured correctly.
Symptom: only one path is used
Look for BGP best-path settings, inconsistent AS-path handling, a missing multipath knob, or one fabric link that is up at Layer 1 but not established at BGP. In small fabrics, asymmetry is often a template drift problem.
Symptom: BGP session does not establish on one interface
Check the physical cabling, LLDP, interface routing mode, MTU and platform-specific requirements for extended next-hop or interface-based neighbors. Also confirm that the peer group is applied to the correct interface; this is where unnumbered designs make drift easy to spot.
Configuration guardrails for production
- Use generated config, not hand-edited snippets. A fabric template should render device ASN, loopback identity, peer group and interface list from inventory.
- Fail closed on route policy. Underlay export should be explicit. If a route is not a loopback needed for transport, it should not leak into the fabric underlay.
- Document failure domains. A link failure should remove one BGP adjacency and reduce ECMP width, not trigger broad reconvergence.
- Keep management separate. Do not use the EVPN fabric as the only way to reach the devices that operate it.
- Collect telemetry early. BGP session count, route count, interface errors and ECMP width should be visible before the first tenant goes live. The Resources page has useful references for routing and monitoring tools.
Small lab bill of materials
You do not need a large rack to learn the control plane. Four virtual NOS instances can teach most of the BGP behavior. Physical switches are useful when you want to validate optics, cables, MTU, interface counters and failure handling. Keep power and noise in mind before buying used data-center hardware.
Practical takeaways
- BGP unnumbered is not magic; it is a way to remove per-link addressing work from a routed fabric.
- The biggest win is operational consistency: fewer unique interface values and easier template validation.
- Do not add EVPN until underlay BGP, loopback reachability and ECMP are clean.
- For AI/GPU clusters and small data-center pods, a boring underlay is a feature, not a compromise.
- If you want more routing context, keep an eye on the BGP Table Watch page and related BGP notes on the blog.
In short: treat BGP unnumbered as a fabric hygiene tool. It will not design your EVPN services for you, but it can make the transport layer easier to automate, easier to audit and easier to repair at 02:00.
Post a Comment