BGP maintenance draining is one of those small operational habits that prevents a planned change from looking like an outage. The idea is simple: before reloading a leaf switch, replacing an optic, moving a firewall uplink, or touching an edge router, make the affected path less attractive, prove traffic has moved, and only then take the disruptive action.
This article shows a practical workflow for data center and service-provider style networks using BGP, EVPN/VXLAN, MPLS VPN or plain Internet edge routing. It is intentionally vendor-neutral, with generic examples you can adapt to IOS-XR, NX-OS, Junos, FRR, Arista EOS or your automation system.
The problem: planned work should not become a routing event
Many maintenance windows still start with an interface shutdown or a device reload. BGP eventually converges, but “eventually” may mean packet loss, microbursts, flow resets, slow reconvergence on remote peers, or noisy alerts. In leaf-spine fabrics this can hit storage replication, AI training jobs and east-west traffic. At the edge it can move Internet ingress too late, after users have already noticed.
The operational goal is not only to keep reachability. The goal is to make the routing change boring: predictable best-path movement, no unexpected withdrawals, no unresolved next-hops, and no surprise asymmetry.
Design: use a drain signal, not a hard failure
A good drain design has two parts:
- A policy signal that makes paths through the maintenance node less preferred while still visible.
- A verification loop that checks control-plane, forwarding-plane and traffic counters before the real change starts.
Common BGP levers include lower local preference, higher MED, AS-path prepend, a provider-specific blackhole/maintenance community, or the well-known GRACEFUL_SHUTDOWN community (65535:0) where your peers honor it. Inside a data center, local preference or a fabric-specific community is usually clearer than AS-path prepending. On Internet edge links, prepend and provider communities are often more practical for inbound traffic engineering.
Implementation workflow
1. Pre-check the topology and blast radius
Before applying any policy, confirm that an alternative path actually exists. In a BGP EVPN/VXLAN fabric this means the remote VTEPs, route reflectors, underlay reachability and ECMP paths are healthy. In an MPLS VPN environment, check PE loopbacks, labels, LSP state and VRF routes. On an Internet edge, check whether the other transit or peering link has enough capacity.
# Generic pre-checks
show bgp summary
show bgp ipv4 unicast <prefix>
show bgp l2vpn evpn route-type 2 | include <mac-or-vni>
show route <prefix>
show forwarding route <prefix>
show interfaces counters errors
show interfaces counters rateDo not skip capacity. A drain that moves 40 Gbit/s of backup or AI storage traffic onto a 25 Gbit/s path will succeed from a routing perspective and still fail operationally.
2. Apply a reversible BGP drain policy
The policy should be narrow, named clearly and easy to remove. A useful pattern is to tag routes from the device under maintenance and then lower preference on neighbors that receive them. The following examples are intentionally generic.
# Example intent, not copy-paste vendor syntax
community-set MAINTENANCE_DRAIN
65535:0
end-set
route-policy SET_DRAIN_OUT
set community MAINTENANCE_DRAIN additive
set local-preference 50
pass
end-policy
router bgp 65000
neighbor-group SPINES_OR_RR
address-family ipv4 unicast
route-policy SET_DRAIN_OUT out
address-family l2vpn evpn
route-policy SET_DRAIN_OUT outFor an edge router, you might apply prepending only to selected public prefixes and leave critical services untouched:
ip prefix-list DRAIN_PUBLIC_PREFIXES permit 203.0.113.0/24
route-map TRANSIT_OUT permit 10
match ip address prefix-list DRAIN_PUBLIC_PREFIXES
set community 65535:0 additive
set as-path prepend 65000 65000 65000
route-map TRANSIT_OUT permit 1000Use documentation prefixes and lab values in templates. Never paste production hostnames, customer names, public IP allocations or tickets into a reusable runbook.
3. Observe convergence before touching the device
After the policy is committed, wait for BGP updates, route reflector propagation and hardware programming. A common mistake is to check only show bgp. That proves the control plane changed, not that the ASIC forwarding table and traffic have moved.
Verification and troubleshooting checklist
Control-plane checks
- The drained node is no longer best path for the target prefixes or VNIs.
- Backup paths are installed, not merely received.
- Route reflectors have propagated the new attributes.
- No unexpected route withdrawals appear in BGP logs.
Forwarding-plane checks
- The FIB points to the expected remaining next-hops.
- ECMP buckets no longer include the maintenance link or device.
- Interface counters show traffic moving away from the drained path.
- There are no adjacency, ARP/ND, label or VXLAN tunnel resolution errors.
Service checks
- Run synthetic pings and TCP probes between representative VLANs, VRFs or tenants.
- Watch packet loss, retransmits, drops and queue depth in your monitoring system.
- For AI clusters, check east-west loss and storage latency, not only default-gateway reachability.
- For backup networks, verify that backup repositories and immutable storage remain reachable through the intended firewall path.
Automation guardrails
This workflow is a good fit for AI-assisted or Ansible-based network automation, but the automation must be conservative. The tool can prepare the change, compare pre/post output and summarize anomalies, while a human still approves the disruptive step.
maintenance_drain:
target: leaf-b
drain_policy: SET_DRAIN_OUT
wait_seconds: 180
required_checks:
- bgp_sessions_established
- backup_paths_installed
- fib_excludes_target
- interface_rate_below_threshold
- synthetic_probes_pass
rollback_if:
- packet_loss_detected
- unexpected_best_path
- bgp_session_flap
- forwarding_unresolvedIf you are building this into a Git workflow, store the runbook, templates and verification commands next to the intended change. The most useful artifact after a maintenance window is not the configuration diff alone; it is the before/after evidence that the drain was safe.
Common mistakes
- Withdrawing routes too early. Prefer de-preference before shutdown, especially when remote convergence is hard to predict.
- Changing all prefixes at once. Drain a small prefix set or one neighbor group first when possible.
- Ignoring inbound traffic. Local preference fixes outbound decisions; inbound Internet traffic may need provider communities or prepend.
- Trusting only the RIB. Hardware forwarding and telemetry must confirm the same result.
- No rollback trigger. Define exactly what makes you remove the drain and stop the maintenance.
Practical takeaways
- Use BGP policy to make a device unattractive before creating a physical failure.
- Keep the drain policy named, narrow and reversible.
- Verify BGP, FIB and traffic counters before reloads or link work.
- For EVPN/VXLAN, include VTEP, VNI and route-reflector checks.
- For Internet edge, remember that inbound and outbound traffic need different tools.
Related reading: start with the Network freak topic index, then browse the Data Center Networking hub, AI Infrastructure and Automation, and the BGP Table Watch page for broader routing context. Useful external references are collected on the networking resources page.
Post a Comment