EVPN/VXLAN MTU Troubleshooting: Why Ping Works but Applications Fail

MTU bugs in an EVPN/VXLAN fabric are frustrating because the control plane usually looks healthy. BGP EVPN is established, VNIs are present, MAC/IP routes are learned, and a normal ping may work. Then a backup job, storage mount, database replication stream or RoCEv2 workload starts moving real payloads and traffic becomes intermittent.

The root cause is often simple: one link, port-channel member, routed subinterface, firewall hop, DCI handoff or host interface cannot carry the larger encapsulated frame. This article gives a practical workflow to prove or eliminate MTU as the issue without guessing.

X Follow @ipexptobe

For related topic hubs, see Start Here, Data Center Networking, AI Infrastructure & Automation, BGP Table Watch and Resources.

Generic derived diagram: VXLAN encapsulation overhead and the point where underlay headroom is required.

Problem: Ping Works, Applications Fail

The classic symptom is not a total outage. It is selective failure:

• Small ICMP pings work between hosts or loopbacks.
• EVPN routes, MAC/IP advertisements and VTEP reachability look correct.
• Large file copies, backups, storage traffic, container image pulls or AI workload transfers stall.
• TCP sessions establish but throughput is poor, retransmits climb, or the application resets.
• RDMA/RoCEv2 or storage paths are more sensitive than ordinary web traffic.

That pattern should immediately trigger a data-plane MTU check. In VXLAN, the packet crossing the underlay is larger than the packet generated by the endpoint. If the fabric was designed for 9000-byte host MTU, the underlay normally needs at least the host MTU plus encapsulation overhead. If the host stays at 1500 bytes, the underlay still needs enough room for a 1500-byte tenant packet after VXLAN encapsulation.

Design Baseline: Decide the MTU Contract

Before troubleshooting, write down the intended contract. A practical fabric standard usually states:

• Host-facing MTU: for example 1500 for general VLANs, or 9000 for storage, backup and AI data paths.
• Overlay MTU: what tenant workloads are allowed to send inside the VNI.
• Underlay MTU: the minimum size on routed leaf-spine links after VXLAN/UDP/IP/Ethernet overhead.
• Boundary MTU: DCI, firewall, WAN, load balancer or service-node handoffs, which are often forgotten.
• Verification command set: the exact DF-bit probes and interface counters accepted as proof.

For a small lab or SMB/private-cloud build, the BOM impact is usually modest: choose NICs, optics/DACs and switches that can support jumbo frames consistently. For example, a pair of 25GbE adapters and DACs might be roughly 250–600 USD/EUR depending on generation and vendor, but the operational requirement is not the price; it is that every hop in the path supports the same MTU policy.

Generic derived diagram: control-plane health does not prove large encapsulated packet delivery.

Diagnosis Workflow

Use a structured path instead of randomly changing MTUs. The goal is to prove where the largest safe packet size changes.

Generic derived workflow diagram based on common data-center troubleshooting patterns.

1. Map the Actual Traffic Path

Do not assume the path is simply “leaf to spine to leaf”. Include:

• host NIC, bond/team, hypervisor vSwitch or container node bridge;
• access port, port-channel and every member link;
• VTEP SVI or routed interface;
• underlay routed links and any subinterfaces;
• firewall, service leaf, DCI, WAN edge or load balancer if traffic leaves the local fabric.

The weakest MTU in this path determines the largest usable packet. A single port-channel member left at a default value can cause intermittent drops when hashing selects that member.

2. Check Interface MTU at Every Layer

Vendor syntax differs, but the checklist is universal:

show interface status | include mtu
show interface ethernet X/Y | include MTU|giants|drops|CRC|pause
show port-channel summary
show interface port-channel N | include MTU|drops
show ip interface brief
show nve peers
show bgp l2vpn evpn summary
show mac address-table vlan <vlan>
show l2route evpn mac-ip all

Look for mismatches between physical interfaces and their bundle, between an SVI and the routed underlay, and between different leaf/spine generations. On some platforms, changing the system jumbo policy requires a reload or affects only newly created interfaces; verify the running state, not just the template.

3. Test with DF Set

Standard ping is not enough. Send a probe with “do not fragment” set and a payload that matches your target. Examples shown as patterns:

# Linux-style pattern
ping -M do -s 1472 <remote-endpoint>      # proves 1500-byte IP packet path
ping -M do -s 8972 <remote-endpoint>      # proves 9000-byte IP packet path

# Network-device pattern
ping <remote-vtep-loopback> size <payload> df-bit source <loopback>
ping <remote-host> size <payload> df-bit source <svi-or-vrf-interface>

Adjust sizes for the platform’s definition of payload versus full IP packet. The exact number matters less than consistency: document the command that proves your design target and use it after every fabric change.

4. Compare Drops and Application Counters

MTU failures rarely announce themselves as “VXLAN MTU problem”. Look for indirect evidence:

• giants, oversized packet drops or L3 MTU drops on routed links;
• port-channel member counters that diverge from the logical interface;
• TCP retransmits, zero-window events or application-level timeouts;
• storage path resets or RDMA queue-pair errors;
• firewall session logs showing “fragmentation needed” or silently dropped oversize packets.

If counters rise only during large transfers, you have a strong signal. If counters are clean, test the host stack next: offloads, driver MTU, bonding mode, overlay in the hypervisor, container CNI settings and any software tunnel layered on top of VXLAN.

Implementation: Guardrails for Changes

The best fix is not a heroic troubleshooting session. It is a pre-change guardrail that refuses to deploy an unsafe VNI, VLAN or workload placement.

Generic derived automation workflow: collect facts, compare path MTU and block unsafe changes before deployment.

A simple automation check can collect interface facts from the fabric and compare the minimum path MTU against the intended tenant MTU plus overhead:

intent:
  tenant: backup-prod
  vni: 10420
  endpoint_mtu: 9000
  required_underlay_mtu: 9100
checks:
  - all_leaf_spine_routed_links_mtu >= required_underlay_mtu
  - all_port_channel_members_match_parent
  - all_border_or_dci_links_mtu >= required_underlay_mtu
  - post_change_df_probe_success == true
result:
  pass: deploy
  fail: block and print weakest_link

This does not need a complicated platform. A scheduled script, Ansible task or Hermes-assisted review can catch obvious drift before the maintenance window. The important part is to make the failure actionable: print the device, interface, configured MTU and required MTU.

Verification Checklist

After changing MTU policy, verify both control plane and data plane:

• EVPN/BGP sessions stable, no unexpected route churn.
• Remote VTEP loopbacks reachable with DF probes at the required size.
• Tenant endpoint-to-endpoint probes pass at the required size.
• Large TCP transfer runs without retransmit spikes.
• Storage, backup or AI data path workload completes at expected throughput.
• No new giants, MTU drops, pause storms or error counters on physical and bundled links.
• Monitoring has an alert for MTU/drop counters, not only BGP neighbor state.

Troubleshooting Shortcuts

• If small ping works but large DF ping fails: binary-search the payload size and test hop-by-hop toward the remote VTEP.
• If only some flows fail: inspect port-channel member MTUs and hash distribution.
• If only one tenant fails: compare SVI, bridge-domain, VRF and host-facing MTU for that VLAN/VNI.
• If only cross-site traffic fails: check DCI, firewall and WAN MTU; the local fabric may be innocent.
• If RDMA/storage fails first: validate host NIC MTU, PFC/ECN policy and offload settings in addition to network MTU.

Practical Takeaways

• EVPN route health does not prove large VXLAN data-plane delivery.
• Define the MTU contract before deploying workloads: host, overlay, underlay and boundary links.
• Use DF-bit probes and counters as proof, not ordinary ping.
• Port-channel member mismatch is a common cause of “random” failures.
• For AI, storage and backup networks, automate MTU pre-checks as part of change review.

MTU troubleshooting is rarely glamorous, but it is one of the highest-value checks in a data-center fabric. A ten-minute verification workflow can save hours of blaming BGP, EVPN, storage arrays or application teams for a simple size mismatch.

Follow Network freak for more networking notes, diagrams and troubleshooting tips.

X @ipexptobe on X