EVPN/VXLAN anycast gateway drift is one of those fabric problems that looks random from the server side: a VM can ping outside its subnet for a few minutes, then traffic stalls, then it recovers after an ARP refresh or a host move. The underlay is clean, BGP EVPN is established, and the SVI IP is configured on both leaves. The real issue is usually smaller: the fabric no longer has one consistent first-hop gateway identity for that VLAN/VNI.
This article gives a practical checklist for finding and fixing anycast gateway MAC, ARP/ND, and EVPN route ownership drift without turning a troubleshooting session into a broad outage. For more related fabric notes, see the Data Center, AI Infrastructure, and Start Here pages.
Problem: when the default gateway is not identical everywhere
In a well-behaved EVPN/VXLAN fabric, every leaf that provides a distributed anycast gateway for a VLAN presents the same gateway IP and the same virtual MAC. Hosts should not care which leaf they are attached to. EVPN type-2 routes should advertise stable MAC/IP bindings, and traffic should be forwarded to the correct local or remote VTEP.
Drift appears when that intent is only partly true. Common examples include:
- one leaf uses a different anycast gateway MAC for the same VLAN/SVI;
- the SVI exists on one leaf but is missing, administratively down, or mapped to the wrong VRF on another;
- the VLAN-to-VNI mapping differs between a vPC/MLAG pair or between racks;
- stale ARP/ND or MAC entries survive after a host migration or gateway change;
- route-target import/export policy makes one rack learn the host route while another rack does not.
Design baseline: define the gateway contract
Before debugging commands, write down the intended contract for the affected bridge domain. This prevents a common mistake: checking that each box has a configuration, instead of checking that every box has the same gateway identity.
The minimum contract for one VLAN/VNI
- Tenant VRF name and L3 VNI.
- VLAN ID and L2 VNI.
- Anycast gateway IP address. Use documentation ranges such as 192.0.2.0/24 in runbooks, never customer addressing.
- Anycast gateway MAC address.
- Route targets for the MAC-VRF and IP-VRF.
- List of leaves where the SVI is expected to be active.
- Host attachment mode: single-homed, vPC/MLAG, or orphan port.
For larger environments and AI clusters, keep this contract in a source-of-truth system and validate it before changes. A small pre-check can compare live state against intent and stop a drift-inducing change before it reaches production.
Diagnosis workflow: prove where the drift starts
Start with the smallest failing unit: one host, one VLAN, one VRF, two local leaves, and one remote leaf if the failure crosses racks. The goal is to avoid clearing large forwarding tables before you know what is wrong.
1. Check the host view
# Linux examples
ip neigh show dev eth0
ip route get 198.51.100.10
arping -I eth0 192.0.2.1
# Windows examples
arp -a
route print
ping 192.0.2.1If the gateway MAC changes between probes while the host stays on the same port, suspect an anycast gateway mismatch or duplicate gateway responder. If only remote traffic fails, keep checking EVPN and VRF state before blaming the host.
2. Compare the SVI and virtual MAC on every expected leaf
show interface vlan 120
show running-config interface vlan 120
show mac address-table vlan 120 | include 0200.0000.1200
show ip arp vrf TENANT-A | include 192.0.2.1Exact command syntax differs by platform, but the comparison is universal. The VLAN, VRF, VNI, gateway IP, and anycast MAC must match on every leaf that can host the workload. A single default MAC on one leaf and an explicit virtual MAC on another is enough to create intermittent behavior.
3. Validate EVPN route ownership
show bgp l2vpn evpn route-type 2 mac-ip 0050.56aa.bbcc
show bgp l2vpn evpn vni 10120
show nve peers
show nve vni 10120 detailLook for the VTEP that owns the host MAC/IP route. If ownership keeps moving without a real host move, check for duplicate attachment, vPC consistency issues, or a loop. If the route is present locally but missing remotely, inspect route targets and BGP EVPN import policy. The BGP Table Watch hub is a useful companion when the control plane is the suspected layer.
4. Check for stale data-plane state
show mac address-table dynamic vlan 120
show ip arp vrf TENANT-A 192.0.2.50
show forwarding vxlan counters
show logging last 200 | include VLAN|VNI|NVE|EVPNStale MAC or ARP entries are often a symptom, not the root cause. Record them first. Then clear only the affected host, VLAN, or ARP entry after the configuration mismatch has been corrected.
Implementation: safe remediation sequence
A safe fix changes the fabric back to one gateway identity and then refreshes stale state in a controlled order.
Normalize configuration first
- Confirm which gateway MAC is the approved standard for the fabric.
- Align SVI IP, virtual MAC, VRF, VLAN, VNI, and route-target settings across all intended leaves.
- Run a configuration diff against peer leaves before committing.
- For MLAG/vPC pairs, run the platform consistency checker before and after the change.
! Vendor-neutral pseudocode: adapt to your platform
interface vlan 120
vrf member TENANT-A
ip address 192.0.2.1/24
fabric forwarding anycast-gateway-mac 0200.0000.1200
no shutdown
vlan 120
vn-segment 10120Refresh only what is necessary
After the config is consistent, use targeted clears instead of a fabric-wide reset:
clear ip arp vrf TENANT-A 192.0.2.50
clear mac address-table dynamic address 0050.56aa.bbcc vlan 120
clear bgp l2vpn evpn route-type 2 mac-ip 0050.56aa.bbcc softSome platforms do not support every targeted command. In that case, prefer the narrowest supported scope and schedule it like a normal change. For a busy storage or GPU cluster, clearing too much at once can create a visible micro-outage.
Verification and troubleshooting checklist
Verification should prove both control plane and data plane health. Do not stop at “the ping works now”.
Post-fix checks
- The gateway MAC is identical on all active leaves for the affected VLAN.
- The host ARP/ND entry resolves to the expected anycast gateway MAC.
- The host MAC/IP EVPN type-2 route is advertised by the correct VTEP.
- Remote leaves import the route into the correct tenant VRF.
- MAC move counters stop increasing.
- Application traffic works in both directions, not only ICMP.
Automation guardrail example
required = {
"tenant": "TENANT-A",
"vlan": 120,
"l2_vni": 10120,
"gateway_ip": "192.0.2.1/24",
"gateway_mac": "0200.0000.1200",
"active_leaves": ["leaf-1", "leaf-2", "leaf-3", "leaf-4"]
}
# Pseudocode workflow:
# 1. collect live SVI/VNI/gateway state from all active leaves
# 2. compare each field with required
# 3. fail the change if any gateway_mac, vrf, or vni value differs
# 4. attach the diff to the change ticket or pull requestThis is a good use case for AI-assisted network automation: let scripts gather deterministic facts, then use an assistant to summarize the mismatches and the operational risk. Keep the final change gate deterministic. More automation references are collected on the Resources page.
Practical takeaways
- Anycast gateway troubleshooting is a consistency problem before it is a packet problem.
- Always compare gateway MAC, SVI, VRF, VLAN, VNI, and EVPN route targets together.
- Do not clear broad MAC/ARP/EVPN state until the configuration drift is fixed.
- Use source-of-truth and pre-change validation to prevent one leaf from becoming the odd gateway.
- Verify with application traffic and EVPN type-2 ownership, not only with a successful ping.
Post a Comment