QoS in Service Provider networks is not only a voice feature or an old WAN trick. It is a practical control system for deciding which packets should survive congestion, how customer services consume shared infrastructure, and how operators can troubleshoot delay, drops, and jitter without guessing. Modern networks often have plenty of bandwidth, but congestion still appears at handoff links, oversubscribed aggregation, encryption gateways, cloud interconnects, mobile backhaul, and during failures when traffic moves to a smaller backup path.
This article gives a field-oriented QoS design and operations checklist for Service Provider and large enterprise WAN environments. The examples use generic devices and documentation addresses only.
Start with the service, not the queue
A common QoS mistake is to begin with a platform command reference and immediately build eight queues. A better starting point is the service contract: what traffic exists, where congestion can happen, and what behavior is required when demand exceeds capacity.
For a Service Provider, typical traffic groups are:
- Network control: routing protocols, LDP/RSVP/SR control traffic, BFD, keepalives, and management protocols that keep the network reachable.
- Real-time: voice, push-to-talk, video conferencing, timing, and other low-latency applications.
- Business-critical data: transactional applications, signaling systems, monitoring, and customer applications with measurable SLAs.
- Standard data: normal web, API, file, and client/server traffic.
- Scavenger or low-priority: backups, bulk replication, software downloads, lab traffic, or traffic suspected during a worm/DoS event.
The goal is not to make every class fast. The goal is to make the failure mode predictable. During congestion, the provider should already know which class is protected, which class is rate-limited, and which class is allowed to lose packets first.
Classification and marking: the trust boundary
QoS only works when traffic is classified consistently. In most designs, the trust boundary should be as close as possible to the traffic source, but not beyond the administrative boundary. A customer can mark every packet as EF, so a provider edge should not blindly trust customer DSCP values unless the service agreement allows it.
A practical ingress policy on a PE might do three things:
- Match known control and management traffic generated by the provider infrastructure.
- Map customer markings into an agreed provider class model.
- Police or remark traffic that exceeds the purchased profile.
Example class model:
Network-Control - CS6/CS7, routing, BFD, infrastructure keepalives
Real-Time - EF, strict latency budget, small percentage cap
Critical-Data - AF31/AF32, assured forwarding
Standard-Data - default forwarding
Scavenger - CS1, backup/bulk/unknown-abusive trafficThe exact DSCP-to-class mapping is less important than consistency. Every PE, aggregation device, and interconnect should agree on the meaning of the marks. If MPLS is used, EXP/TC bits also need a documented mapping so packets do not change treatment when they enter a label-switched path.
Policing versus shaping
Policing and shaping are often confused because both use a rate value. Their behavior is very different. A policer enforces a rate by dropping or remarking traffic that exceeds the token bucket. A shaper buffers bursts and releases packets at a configured rate. In simple terms: police when you must enforce a contract; shape when you want to adapt to a slower downstream link.
On a customer-facing PE interface, policing is useful for CIR/EIR enforcement. If a customer buys 200 Mbps on a 1 Gbps port, the provider may allow committed traffic, mark excess as lower priority, or drop traffic above the contract. At a branch WAN edge, shaping is usually better because the router knows the access link is slower than the physical interface and can queue packets intelligently before the carrier drops them.
Watch for double punishment. If customer traffic is shaped at the CE and then harshly policed at the PE with a smaller burst size, TCP may behave poorly and voice may suffer intermittent drops. Align token bucket sizes and document whether excess traffic is dropped or remarked.
Queuing: protect real-time traffic without starving the rest
Queuing decides what happens after congestion already exists. A low-latency or priority queue is useful for voice and other real-time flows, but it must be capped. An unlimited priority queue can starve other classes, especially during a malformed traffic event where too many packets are marked as EF.
A conservative starting point is to reserve a small percentage for strict priority traffic, allocate assured bandwidth to critical data, and leave enough capacity for default traffic. For example on a 1 Gbps logical service:
Real-Time priority: 100 Mbps maximum
Critical-Data guarantee: 250 Mbps
Standard-Data: remaining bandwidth
Scavenger: minimum bandwidth, drop firstThis is not a universal template. A mobile backhaul interface, a cloud interconnect, and an enterprise VPN service can have very different traffic mixes. The key operational point is that the queue design must match measured traffic, not assumptions from an old standard profile.
Where QoS matters most in an SP path
QoS configuration on one router is rarely enough. The effective behavior is the minimum quality across the entire path. Focus on these locations:
- Customer edge to provider edge: classification, trust boundary, policing, and customer SLA enforcement.
- Aggregation uplinks: oversubscription appears here when many customers or access rings converge.
- Core links after failure: a backup path may be smaller or more loaded than the primary path.
- MPLS or SR transport: EXP/TC treatment must preserve the intended forwarding class.
- Internet, cloud, or NNI handoff: markings may be rewritten, ignored, or mapped into a partner model.
During design reviews, draw the packet path and write the QoS action at each hop: classify, mark, police, queue, shape, remark, or trust. Missing actions are often where the real outage hides.
Troubleshooting symptoms and quick checks
When an application team reports "the network is slow," QoS issues usually appear as delay variation, packet drops under load, or asymmetric behavior. Use counters before changing policy.
- Class counters not increasing: the match statement is wrong, the traffic is marked differently than expected, or classification happens on the wrong interface direction.
- Drops in the priority queue: the real-time cap is too low, traffic is over-marked as EF, or an attack is entering the priority class.
- Drops only after failover: the backup path has lower capacity or a different queue policy.
- Customer says they mark correctly, provider sees default: DSCP may be reset by a firewall, tunnel endpoint, SD-WAN appliance, or interconnect.
- TCP throughput is unexpectedly low: policer burst sizes may be too small, causing frequent micro-drops.
Useful verification commands vary by vendor, but the workflow is universal: check interface utilization, check per-class packet and drop counters, verify DSCP/EXP mappings, confirm the active policy direction, and repeat during the problem window. QoS counters collected at idle time prove very little.
Operational guardrails
QoS can become dangerous when every service asks for a special class. Keep the class model small enough for humans to operate. Four to six operational classes are often easier to validate than a complex matrix of ten or more queues.
Document these items:
- DSCP and MPLS TC/EXP mapping table.
- Which interfaces trust customer markings and which rewrite them.
- Policer rates, burst values, and exceed actions.
- Queue guarantees and priority caps.
- Expected behavior during link failure and maintenance.
Also monitor QoS counters. A class with constant drops is either underprovisioned, incorrectly classified, or carrying traffic it was never designed to carry. A class with zero packets for months may be obsolete or matched in the wrong place.
Lab idea: build congestion on purpose
A small lab can teach more than a long slide deck. Build three Linux hosts or virtual routers connected through a bottleneck link. Generate one UDP stream to represent voice, one TCP stream for normal data, and one bulk transfer. Mark the flows with different DSCP values, then apply a queue policy on the bottleneck interface.
Test three scenarios: no QoS, priority queue without a cap, and priority queue with a cap plus a scavenger class. Measure packet loss, jitter, and TCP throughput. The lesson is simple: QoS does not create bandwidth, but it can decide which application remains usable when bandwidth is not enough.
Summary
QoS in Service Provider networks is a design, enforcement, and troubleshooting discipline. Start with service requirements, define a small class model, place the trust boundary carefully, map markings consistently across IP and MPLS/SR transport, and validate behavior with counters during congestion. The best QoS policy is not the most complex one; it is the one the operations team can explain at 03:00 during an incident.
Related reading
Discussion
If you have worked with QoS in a real network or lab, feel free to share your experience, design notes, or troubleshooting lessons in the comments.
Source inspiration: Cisco Press End-to-End QoS Network Design and 7750 SR OS Quality of Service Guide.
Post a Comment