EVPN/VXLAN ARP suppression troubleshooting is one of those data center tasks where the symptom can be misleading. A packet capture may show an ARP reply, the default gateway SVI may look up, and the BGP EVPN sessions may all be established, but the workload still cannot reach a peer in another rack. The reason is simple: ARP suppression moves part of first-hop discovery from flood-and-learn behavior into a distributed control-plane cache. If that cache contains a stale or incomplete MAC/IP binding, the fabric can answer confidently and still be wrong.
This article gives a practical workflow for diagnosing that situation without exposing any customer-specific details. It is written for EVPN/VXLAN fabrics using anycast gateways, distributed bridging/routing and BGP EVPN route type-2 advertisements. It also fits the broader Data Center Networking and AI Infrastructure & Automation topics, because GPU clusters, storage networks and virtualized platforms are especially sensitive to small fabric inconsistencies.
If you are new to these topics, start from the Start Here networking topics page and keep the Resources page handy for protocol references.
The problem: ARP replies are present, but traffic still fails
Classic Layer 2 troubleshooting starts with a broadcast ARP request and a unicast ARP reply. EVPN/VXLAN with ARP suppression changes that behavior. A leaf switch may answer locally on behalf of a remote endpoint when it has learned the endpoint's MAC/IP mapping through EVPN. That is excellent for scale because it reduces broadcast in the overlay, but it adds a new failure mode: the local leaf can suppress the broadcast and return information that no longer matches reality.
Common symptoms include:
- A host receives an ARP reply for a remote host, but ICMP, TCP or storage traffic fails.
- The same source host can reach some racks or VNIs but not one specific rack, VLAN or tenant segment.
- Clearing ARP on the server helps temporarily, then the problem returns after mobility, failover or vMotion/live migration.
- One leaf has the correct MAC/IP binding while another leaf has an older binding for the same IP address.
- Flooding works when suppression is disabled in a test window, but the routed or bridged service fails with suppression enabled.
Design background: what must be consistent
For ARP suppression to work reliably, the fabric needs agreement across several layers. Do not treat it as only an ARP feature. It depends on endpoint learning, BGP EVPN advertisements, tenant import/export policy, VNI-to-VLAN mapping and underlay reachability.
Endpoint learning
The ingress leaf must know the local endpoint and advertise the right MAC/IP pair. If a server has moved, changed bonding mode, failed over to a new NIC or reused an IP address, the old leaf may still have stale state. In fabrics connected to hypervisors, orphan ports, firewalls or load balancers, this is often the first place to check.
EVPN route type-2 propagation
The remote leaf learns MAC/IP information through EVPN route type-2. Verify that the route exists, has the expected route distinguisher and route target, points to the correct VTEP, and is not hidden by policy. This is where the BGP Table Watch habit helps: if the control plane changed, endpoint reachability can change even when links are up.
Anycast gateway behavior
Anycast gateways make first-hop routing resilient, but they also mean multiple leaf switches answer using the same gateway MAC/IP. Confirm that gateway configuration, VLAN/VNI mapping, VRF membership and suppression settings are uniform on every leaf that hosts the segment.
Diagnosis workflow: follow the binding
The fastest workflow is to follow one source IP, one destination IP and one VNI end to end. Avoid jumping between many hosts at once. Build a small table with the source leaf, destination leaf, source MAC, destination MAC, VNI, VLAN, VRF and expected VTEP.
1. Start at the host and adjacent leaf
# On the source host
ip neigh show DESTINATION_IP
arp -an | grep DESTINATION_IP
# On the adjacent leaf, syntax varies by NOS
show ip arp vrf TENANT | include DESTINATION_IP
show mac address-table dynamic vlan VLAN_ID | include DESTINATION_MAC
show nve peers
show nve vni VNI_ID
Look for a mismatch between the host ARP cache, the leaf ARP/ND suppression cache and the EVPN endpoint table. If the host has MAC bb:bb but the leaf is suppressing with cc:cc, do not continue into MTU testing yet; fix the binding source.
2. Check EVPN route type-2 on multiple leaves
show bgp l2vpn evpn route-type 2 | include DESTINATION_IP
show bgp l2vpn evpn route-type 2 mac-ip DESTINATION_MAC DESTINATION_IP
show bgp l2vpn evpn route-type 2 rd * vni VNI_ID
On a healthy fabric, all relevant leaves should agree on the next-hop VTEP for the destination endpoint. If Leaf A points to VTEP-1 and Leaf C points to VTEP-3 for the same IP, you have either stale state, duplicate addressing, asymmetric import policy or a mobility sequence that did not converge cleanly.
3. Validate VNI, route-target and tenant policy
ARP suppression can only answer correctly when the bridge domain is mapped consistently. Check for:
- VLAN mapped to the same L2 VNI on all participating leaves.
- VRF mapped to the correct L3 VNI where inter-subnet routing is involved.
- Route targets imported and exported consistently.
- No local route-map, community filter or tenant policy suppressing type-2 routes.
- No duplicate anycast gateway MAC configuration across different tenant domains.
4. Only then test data plane
After the control plane agrees, test the VXLAN data plane: underlay reachability between VTEPs, MTU, UDP 4789 counters, ECMP hashing and drops on the uplinks. A jumbo-frame or ECMP issue can look similar, but if the ARP cache is already wrong, data-plane testing will waste time.
Implementation and configuration checks
Exact syntax differs between vendors, but the intent is portable. Add these checks to your pre-change and post-change workflow:
- For each critical VLAN/VNI, collect the local endpoint table from all leaves.
- Compare EVPN type-2 routes for duplicate IPs and duplicate MACs.
- Verify suppression is enabled only where the platform has reliable MAC/IP learning.
- Document which segments contain firewalls, load balancers, storage arrays or appliances that may use virtual MACs.
- For hypervisor clusters, align live-migration procedures with MAC mobility timers and endpoint aging.
A small AI-assisted automation job can help here. Export sanitized command output, normalize it into a table and ask the assistant to highlight contradictions: same IP with multiple MACs, same MAC behind multiple VTEPs, VLAN/VNI mismatches and missing route targets. Keep the automation read-only unless the change has a human approval step.
Verification checklist
Before closing the incident, verify from more than one vantage point:
- The source host ARP/ND entry matches the intended destination MAC.
- The source leaf suppression cache matches the EVPN MAC/IP route.
- The EVPN route type-2 next-hop is the destination leaf VTEP, not an old location.
- The destination leaf has the endpoint as local and active.
- Route-target import/export is correct for the tenant and VNI.
- VXLAN underlay reachability and MTU are clean between the VTEPs.
- Application testing confirms the real port/protocol, not only ping.
Troubleshooting traps
Clearing only the server ARP cache
This may hide the issue for a few minutes. If the leaf still has a stale suppression entry or stale EVPN route, the host will relearn the same wrong answer.
Blaming the route reflector too early
Route reflectors distribute what they receive. Check whether the advertising leaf is sending the wrong MAC/IP pair before assuming the RR is at fault.
Ignoring duplicate IP detection
Duplicate addressing in a tenant VLAN can create a very convincing ARP suppression problem. Search the whole fabric for the IP and MAC, not only the two racks involved in the ticket.
Testing during endpoint mobility
Live migration, firewall failover and storage controller failover can legitimately move a MAC. Correlate timestamps with endpoint mobility logs and aging timers before deciding that the fabric is unstable.
Practical takeaways
- ARP suppression is a control-plane optimization, so troubleshoot the MAC/IP binding before deep-diving into packet drops.
- Follow one endpoint pair across host ARP, leaf cache, EVPN type-2 routes, VNI mapping and data-plane counters.
- Stale or contradictory MAC/IP information is more useful evidence than a generic statement that “BGP is up”.
- For AI, storage and virtualization fabrics, automate read-only consistency checks for critical VNIs before maintenance windows.
- Keep diagrams generic and sanitized: the useful pattern is the binding workflow, not the customer-specific topology.
Related reading: Data Center Networking, AI Infrastructure & Automation, BGP Table Watch and the Resources page.
Post a Comment