A connection can establish successfully and still deliver a painfully slow file transfer. The useful interview question is not simply “Can the firewall pass TCP?” but “Where is this particular transaction losing time?”
Short answer: First separate slow connection setup, slow application response and slow bulk transfer. Then reproduce one flow and compare simultaneous captures on both sides of the firewall, together with interface and policy-counter changes; Cisco recommends this method to distinguish loss or delay before, inside and after an ASA.[1] Investigate path MTU and TCP receive-window limits as well as the firewall itself.[2][3]
Why can client-to-server traffic through a firewall be slow?
Packet loss and delay can reduce TCP performance even when the connection remains established.[1] A PMTU black hole can allow the handshake and small requests while preventing large data packets from completing the transfer.[2] A receive window that is too small for the bandwidth-delay product can also limit a single connection, even without an overloaded link.[3]
Treat the following as a troubleshooting workflow, not a claim that every firewall has the same forwarding architecture or commands. The Cisco reference describes ASA-specific behavior, including inspection-related stream coalescing; do not apply its historical MSS defaults or IPS-module behavior to unrelated products.[1]
Step 1: define what “slow” actually means
Record the client, destination, port, direction, test time, transferred bytes and elapsed time. Use a repeatable test object rather than comparing two unrelated downloads. Ask whether all clients are affected, whether uploads and downloads differ, and whether the symptom started after a change.
Use this decision table to choose the next measurement. These are diagnostic hypotheses, not automatic root-cause verdicts.
| Observed symptom | Investigate next | Evidence to collect |
|---|---|---|
| Long delay before the first connection attempt | Name resolution, proxy selection, client behavior | DNS timing and client-side trace |
| Repeated SYNs before establishment | Reachability, policy and return path | Both directions of the handshake |
| Fast handshake, long pause after the request | Application, TLS/proxy processing, backend dependencies | Request/response timing plus server logs |
| Small exchanges work, large transfers stall | Path MTU and required ICMP delivery | Packet size at failure and PMTU feedback |
| Transfer starts, then repeatedly slows | Loss, congestion, policing, inspection or endpoint limits | Paired captures and counter deltas |
| One flow is slow, several flows together are faster | Per-flow window/RTT or processing limits | Single-flow versus aggregate results |
Do not call an application “healthy” merely because the port opens. Keep connection establishment and post-handshake performance as separate questions.
Step 2: follow the data and its acknowledgments
A useful conceptual capture layout is:
Client -- A: firewall ingress -- [policy / inspection] -- B: firewall egress -- Server
Client <-- A: reverse egress <-- [session handling] <-- B: reverse ingress <-- ACKs
Cisco explicitly recommends simultaneous, bidirectional captures on the affected ASA interfaces and statistics before and after the test.[1] For your investigation, start before the connection opens and stop after one reproducible failure. Include the handshake because the TCP Window Scale option is exchanged in SYN-bearing segments, not renegotiated halfway through the connection.[3]
Compare the same data ranges, acknowledgments and timing at A and B. Account for NAT when matching addresses. If the device proxies connections, analyze the two transport legs rather than expecting identical end-to-end packet identities.
Use this evidence ladder:
- Not visible at A: extend the investigation toward the sender and upstream path.
- Visible at A, missing at B: check the firewall session, policy, drop reasons and whether the capture actually observes the forwarding path.
- Visible promptly at both A and B, missing at the server: extend the investigation beyond the firewall.
- Data delivered, acknowledgment absent on the reverse path: examine the receiver and return direction separately.
- Both directions cross promptly, application remains slow: correlate endpoint and application timing before changing firewall policy.
This implements the before/inside/after distinction in Cisco's capture-analysis methodology.[1] A missing packet in one trace is a lead, not final proof: first exclude an incorrect filter, capture loss or an unobserved accelerated path. For cross-host timestamp comparisons, validate clock synchronization before attributing a small timing difference to forwarding delay.
Step 3: check loss, errors and inspection with counters
Compare counter changes during the same test, rather than treating an old cumulative error count as a current fault. Cisco's ASA method calls for interface and service-policy statistics; it also identifies speed/duplex mismatch and inspection-related stream handling as possible causes.[1]
Illustrative read-only ASA checks, subject to release and platform support:
show interface
show service-policy
Look for an error or drop increase that aligns with the affected flow and test interval. Also review platform-specific dataplane utilization, queues, session capacity and configured rate limits. A low management-plane CPU reading alone is not enough evidence to clear the forwarding path.
Do not disable security globally as the first test. If inspection is a supported hypothesis, use an approved, narrowly scoped exception in an isolated test or maintenance window, keep the test otherwise unchanged, and restore the policy immediately afterward. A faster result under a different policy is a clue to investigate, not permission to leave protection removed.
Step 4: recognize a path MTU black hole
RFC 2923 describes the classic signature: small packets and some interactive connections succeed, but bulk transfer fails when larger packets cannot traverse the path and the sender does not receive the necessary ICMP feedback.[2] Therefore, successful ping with a small payload does not prove that the path can carry the transfer's larger packets.[2]
Check the negotiated MSS, actual data-packet sizes, tunnel overhead and delivery of the relevant PMTU error messages. Test both directions. Treat a failed large probe cautiously: filtering or a different probe path can also explain failure, so correlate it with the affected TCP flow.
Avoid setting an arbitrary low MSS as a universal cure. Cisco warns that changing MSS without considering encapsulation and fragmentation risks is unsafe as a blanket performance fix.[1] Correct the demonstrated path-MTU or feedback problem, and use a justified MSS adjustment only where appropriate for the actual platform and path.
Step 5: understand TCP window size versus bandwidth
Window scaling exists because the original TCP receive-window field can restrict throughput on paths with a large bandwidth-delay product.[3] The receiver's advertised window and the sender's congestion window are distinct; scaling the receive-window field does not scale the congestion-control window.[3]
For a simple diagnostic estimate, suppose an illustrative connection has an effective receive window of 65,536 bytes and a 40 ms RTT. Its receive-window-limited ceiling is approximately:
window-limited rate = receive-window bytes × 8 / RTT seconds
= 65,536 × 8 / 0.040
= 13,107,200 bit/s
≈ 13.1 Mbit/s
To keep a 1 Gbit/s path busy at 40 ms RTT:
bandwidth-delay product = 1,000,000,000 × 0.040 / 8
= 5,000,000 bytes in flight
These are calculated examples, not measured firewall results. The estimate illustrates the window/RTT limitation described in RFC 7323; it is not a throughput guarantee.[3] Congestion control, loss, application delivery and other bottlenecks may reduce achieved throughput further. Do not assume that increasing link speed or a socket buffer alone will fix the transfer.
Investigate repeated small or zero receive-window advertisements at the advertising endpoint. Do not confuse the receiver asking the sender to pause with proof that the firewall dropped packets. Similarly, a large advertised window only permits more data in flight; it does not prove the sender can or will generate it.
A concise interview answer
“I first define whether the problem is connection setup, application response or bulk throughput. I reproduce a single flow, capture both directions on both firewall sides and compare interface and policy counters over the same interval. That lets me localize loss or delay instead of blaming the firewall from one slow download.”[1]
“I then check whether large packets fail despite a working handshake, which suggests a possible PMTU black hole, and whether the receive window is adequate for the RTT and target bandwidth.”[2][3]
“I correlate the evidence with inspection, rate limits and endpoint health, change one thing at a time under approval, and repeat the identical test to verify the fix.”
Common interview pitfalls
- “Ping works, so the network is fine.” Small-packet success can coexist with a PMTU black hole.[2]
- “The handshake completed, so throughput must be fine.” Connection establishment does not demonstrate that larger data packets can cross the path.[2]
- “A retransmission proves the firewall dropped it.” Localize the missing or delayed data using paired captures first.[1]
- “Window scaling and congestion control are the same.” They are different mechanisms.[3]
- “The datasheet rate is what every application should get.” Demand a test matching your traffic and enabled features rather than substituting an unrelated benchmark.
- “Disable IPS, then close the incident.” Require an approved remediation that preserves the intended security posture.
Suggested lab and verification checklist
This is a proposed lab, not an executed appliance test. Use isolated clients and servers with a firewall between them. Establish a baseline, introduce one controlled impairment at a time, and identify which observation changes.
- Record the exact test, direction, policy and elapsed time.
- Capture the handshake and data on both sides, with authorization and minimal scope.
- Compare counter deltas, not just screenshots of lifetime totals.
- Test a constrained path MTU separately from a latency or receive-window constraint.
- Remove the impairment and repeat the baseline test.
- Confirm the application improves and required security controls remain enabled.
Packet captures can contain credentials or private payloads. Restrict access, retention and sharing; sanitize evidence before attaching it to a public discussion.
Related reading
- TCP 3-Way Handshake Explained: TCP Flags and Firewall Troubleshooting
- Stateful vs Stateless Firewall Explained
- Zone-Based Firewall: Class-Map, Policy-Map and Service-Policy
Summary
A strong answer localizes the bottleneck: define the symptom, compare both packet directions, correlate counter changes, then test the MTU, window or processing hypothesis supported by the evidence. Do not confuse a working session with a healthy transfer.
Preparing for a network engineering interview? Practice explaining which observation would prove or disprove your next hypothesis before naming a command or proposing a firewall change.
Post a Comment