RoCEv2 troubleshooting is hard because the failure often looks like an application problem: GPU jobs stall, checkpoint writes take longer, a storage node looks slow, or one rack suddenly drags down the rest of the pod. The network may show no classic packet loss on a normal interface graph, while the real issue is hiding in priority queues, ECN marking, PFC pause frames, NIC classification, or a backup flow accidentally sharing the lossless class.
This practical checklist is for small and mid-size AI clusters, data-center labs and private GPU pods where RoCEv2 is used for storage, distributed training or low-latency east-west traffic. It complements the AI Infrastructure hub, the Data Center Networking hub, and the broader Start Here networking topics page.
Problem: lossless Ethernet can amplify small mistakes
RoCEv2 runs RDMA over UDP/IP. That makes it routable and attractive for leaf-spine designs, but it also means congestion behavior must be deliberate. A normal TCP flow will reduce its rate after loss. RDMA applications expect very low loss and often depend on congestion notification and careful buffering before loss happens.
The most common operational mistake is enabling lossless everywhere. Priority Flow Control (PFC) is useful when scoped tightly, but dangerous when applied broadly. If backups, package downloads, logging, VM migrations or internet traffic share the same PFC-enabled class as RDMA traffic, a congested receiver can send pause frames that propagate pain into unrelated workloads.
Typical symptoms
- GPU training jobs periodically stall during checkpoint or data-loader phases.
- Storage latency spikes without obvious link saturation on standard graphs.
- Switch interface counters look clean, but class queue or buffer counters are not clean.
- PFC pause frames appear on links that should not carry the RDMA class.
- ECN marks are always zero, even during obvious congestion, or explode during backup windows.
Design: build a small failure domain first
A practical RoCEv2 design starts with traffic classification. Pick one priority for RDMA data, map it consistently from the host NIC to the access/leaf switch and across the fabric, and keep everything else out. Management traffic and control-plane traffic need protection, but they do not need to be in the same lossless class as RDMA payload.
For a small pod, I prefer a staged model:
- Best effort for normal traffic. Backups, OS updates, package mirrors, logging and user access stay out of PFC.
- One RDMA class. RoCEv2 data gets a predictable DSCP/PCP value, ECN marking and PFC only where required.
- Protected control/management. Routing protocols, SSH, out-of-band access and monitoring use separate queues and rate limits.
- Rack or pod boundary. Test one rack before enabling fabric-wide behavior.
Implementation: a safe configuration workflow
The exact syntax differs by switch and NIC vendor, but the workflow is consistent. Do not start by enabling PFC everywhere. Start by proving classification, then enable congestion signaling, then enable PFC only on the class that truly needs it.
1. Prove host marking before touching PFC
On each server, verify the RDMA application or NIC policy marks the intended traffic. If the host does not mark correctly, the switch will put the flow into the wrong queue and the rest of the design becomes guesswork.
# Linux examples vary by NIC/driver; adapt to your platform
ip link show
ethtool -S <interface> | egrep -i 'prio|pfc|pause|ecn|rdma|drop'
rdma link show
# Packet capture sanity check on a test flow
# Confirm DSCP/ECN bits and UDP destination behavior for RoCEv2 traffic.
tcpdump -i <interface> -vv 'udp'2. Map one RDMA class through the fabric
Use a simple mapping document: application or NIC policy → DSCP/PCP → switch ingress class → queue → ECN threshold → PFC priority. Keep that mapping in Git next to your network change record. If an incident happens later, you need to know whether a paused queue is expected or leaked.
# Vendor-neutral intent example, not copy/paste configuration
class RDMA-DATA
match dscp 26
queue rdma-lossless
ecn enable threshold low 50% high 70%
pfc enable priority 3
class DEFAULT
pfc disable
ecn optional
class CONTROL-MGMT
priority protected
pfc disable3. Enable ECN before depending on pause
ECN should mark congestion before the queue is in danger. If ECN thresholds are too high, the first visible symptom may be drops or pause storms. If thresholds are too low, endpoints may throttle too aggressively. The right answer is platform dependent, so baseline under real workload, not only under synthetic iperf tests.
4. Scope PFC narrowly
PFC is a tool, not a design philosophy. Enable it on the RDMA priority only, on interfaces that carry the RDMA class, and with a watchdog or equivalent protection where available. Never assume all 100/200/400GbE ports in the rack require identical lossless treatment.
Verification: counters that matter more than interface utilization
A 40% utilized uplink can still microburst into a shallow queue. A graph showing no interface errors does not prove the RDMA class is healthy. During rollout, collect queue-level and host-level evidence before, during and after the change.
| Signal | Healthy pattern | Bad pattern |
|---|---|---|
| ECN marks | Appear during real congestion and then settle | Always zero, or spikes exactly when backups start |
| PFC pause Rx/Tx | Rare, bounded, tied to known bursts | Continuously increasing, especially on one link |
| RDMA queue drops | Zero or explained by a test | Any unexplained drop during production workload |
| No-buffer drops | Zero in steady state | Microburst or incast pressure not visible in Mbps graphs |
| Host counters | No RDMA errors, no unexpected retransmit symptoms | CQ errors, NIC pause spikes, application retry storms |
Troubleshooting workflow: isolate classification, congestion and endpoint behavior
When a GPU or storage team reports slowness, resist the urge to randomly tune thresholds. Use a repeatable path.
Step 1: confirm the blast radius
Is the problem one host, one rack, one storage target, one spine path, or the whole pod? If only one rack is affected, compare leaf counters and NIC counters between a good rack and a bad rack. If all racks are affected during a specific time window, correlate with backups, data ingest, checkpoint schedules or maintenance jobs.
Step 2: verify traffic is in the intended class
Many RoCEv2 incidents are classification incidents. Check whether DSCP is rewritten by a hypervisor, storage gateway, firewall, tunnel endpoint or NIC profile. If the RDMA flow falls into best effort, you may see drops. If best-effort backup traffic falls into the RDMA class, you may see pause storms.
Step 3: look for pause source, not only pause victim
A port receiving pause frames is not always the root cause. Identify who transmits pause and why. A receiver with insufficient buffers, a slow storage node, a single oversubscribed uplink, or a misclassified elephant flow can make healthy senders look guilty.
Step 4: check ECN and endpoint reaction together
ECN marks alone do not fix congestion; endpoints must react. If switches mark but NIC or driver congestion control is misconfigured, queues still grow. If endpoints react too strongly, throughput collapses. Capture both sides: switch ECN marks and host/NIC congestion-control counters.
Automation angle: make the pre-check boring
This is a good place for AI-assisted network automation, but with guardrails. A small script can collect queue counters, PFC counters, ECN marks, NIC counters and current QoS policy before a change. An AI assistant can summarize differences between baseline and post-change output, but it should not invent missing counters or approve a change without thresholds.
A useful automation checklist:
- Save the intended DSCP/PCP/PFC mapping in a version-controlled YAML file.
- Collect pre-change counters from every leaf and a sample of hosts.
- Run a small RDMA test flow and a normal backup flow to prove separation.
- Fail the change automatically if PFC appears on non-RDMA classes.
- Generate a rollback note with exact interfaces and policies touched.
For adjacent automation ideas, see AI Infrastructure and Automation, and for external tools and references keep the networking resources page handy.
Practical takeaways
- Do not make the whole data center lossless. Make the RDMA class predictable and narrow.
- Prove host marking before enabling PFC; otherwise you are protecting the wrong traffic.
- ECN, PFC and endpoint congestion control must be verified as one system.
- Queue counters matter more than average interface utilization during RoCEv2 incidents.
- Keep backups and bulk transfers out of the RDMA class, especially in small AI clusters.
- Stage changes by rack or pod and define rollback triggers before the change starts.
RoCEv2 can work very well in a practical AI fabric, but it rewards disciplined operations. Treat lossless behavior as a scoped exception, measure it continuously, and make classification boring enough that troubleshooting starts with evidence instead of guesswork.
Post a Comment