A firewall changing from passive to active is not the same as an application surviving a failure. Use this Palo Alto firewall HA failover test plan to separate role election, session continuity, NAT behavior, routing recovery and user-visible service restoration.
The durable asset below is a copyable acceptance matrix and change-window checklist for an active/passive pair. It is a proposed engineering workflow, not a report of a lab run. Adapt it to your appliance or VM platform, PAN-OS release, management method and application recovery objectives; it does not cover active/active HA or cloud-provider failover integrations.
The problem: the dashboard is green, but users still disconnect
Do not use configuration synchronization as proof that every connection will survive. Palo Alto Networks documents configuration synchronization and runtime synchronization separately, with the session table synchronized over HA2 and several explicit exceptions.[1]
For active/passive HA, ICMP and host sessions are not synchronized; the vendor also documents that decrypted SSL sessions do not support HA synchronization after failover.[1] Therefore, do not make “zero lost pings” your sole acceptance criterion, or promise uninterrupted decrypted HTTPS sessions merely because ordinary TCP sessions survived.
Start with a service-level question: after losing the active firewall, can a user complete a real transaction within the agreed interruption budget, without a policy bypass or manual intervention?
Define the test boundary before touching HA
Use a generic topology with test clients on one side, a representative application on the other, and independent management access to both peers. Include the real upstream and downstream switching and routing paths in your observation scope. A local firewall test cannot certify an untested WAN circuit or remote application dependency.
Select a small set of authorized synthetic workloads. Prefer a read-only application transaction or dedicated test account, rather than a production order, payment or destructive database action. Collect these separately:
- Established flow: start before the event and verify whether the same connection continues.
- New connection: repeatedly start fresh connections before, during and after the event.
- Application transaction: validate expected content or a safe application operation, not just an HTTP response code.
- Negative control: send an approved test that policy must deny, and verify it remains denied after takeover.
Write the expected results before the test. “Service recovered” and “existing session survived” are different outcomes; a retrying application may mask a connection reset.
Pre-change checklist: stop if these gates fail
Configuration and peer readiness
Palo Alto Networks states that only committed configurations synchronize through normal HA configuration synchronization; configuration still in the commit queue is not synchronized.[1] Its reference also states that configuration pushed from Panorama is not synchronized between the firewalls through HA.[1]
Accordingly, verify the intended device-group and template deployment on both managed peers, rather than treating one successful Panorama push as a pair-level acceptance result. For local changes, confirm the intended committed state on both peers and review any mismatch before proceeding.
- [ ] Approved maintenance window, application owner and incident contact recorded.
- [ ] Current configuration exported and a known-good recovery procedure available.
- [ ] Independent management or console access tested for each peer.
- [ ] Intended software, content, licenses and platform compatibility checked against your release documentation.
- [ ] No unexplained suspended, non-functional or synchronization-error condition.
- [ ] HA links, configured backups and session synchronization checked.
- [ ] Preemption behavior and the intended final active peer recorded.
- [ ] Both peers can reach required management dependencies through their intended paths.
Management interface settings, HA settings, licenses and several other settings do not synchronize automatically between active/passive peers.[1] Treat them as a per-device checklist, not as an assumed consequence of a green configuration status.
Session readiness is a separate gate
The vendor identifies disabled state synchronization and a down HA2 link or connection as reasons sessions do not synchronize.[1] Its setup guide explains that session synchronization lets the secondary match packets against synchronized sessions; without it, recreating sessions can introduce latency and dropped connections.[2]
Generate a representative, ordinary transit TCP flow and inspect the relevant session information on both devices using the tools supported by your release. Match the endpoints and translation details; do not demand identical global session counts or assume every session class should be mirrored.
Do not start a planned switchover with unexplained HA2 degradation. Restore synchronization first, then establish fresh baseline flows. A test after deliberately disabling synchronization is a different experiment and needs its own risk approval.
Copyable firewall HA acceptance matrix
The following is an original test template. “Budget” means a service-specific target agreed before execution, not a vendor guarantee. Record actual results in the final column; the blank fields are intentional.
| Test | Probe and evidence | Acceptance criterion | Actual result |
|---|---|---|---|
| Planned role change | HA event timestamps plus application probes | Intended standby takes over; service recovers within budget | Not run |
| Existing non-decrypted TCP | Long-lived test connection; client/server logs; targeted session inspection | Continuity meets the application's requirement; any reset explicitly recorded | Not run |
| New TCP/HTTPS connections | Fresh connections with response validation | New transactions recover within budget | Not run |
| Decrypted HTTPS | Authorized decrypted test application; retry and TLS observations | Recovery/reconnection meets the agreed target; do not assume session preservation | Not run |
| Source NAT egress | Dedicated outbound transaction; translated tuple and return traffic | Required translation behavior and return path verified | Not run |
| Published service / destination NAT | Probe from an authorized external location | Correct service reachable through intended rule and translation | Not run |
| Dynamic routing | Neighbor state, selected routes and actual forwarded traffic | Required destinations recover; no unintended path or route leak | Not run |
| IPsec service | Real traffic inside the tunnel plus relevant SA state | Application traffic resumes within budget; control-plane and data-plane states recorded separately | Not run |
| Required deny rule | Approved forbidden test flow | Still blocked after takeover; correct security evidence retained | Not run |
| Monitoring and logging | HA event, alert receipt and new-active traffic logs | Operators detect event; security visibility remains available | Not run |
| Controlled restoration | Repeat positive and negative probes | Intended final roles stable; synchronization restored | Not run |
The exception for decrypted SSL sessions comes from the synchronization reference.[1] That reference also distinguishes unsynchronized IKE phase-1 SAs from synchronized IPsec phase-2 SAs, so an “IPsec up” indicator alone should not replace a real traffic test.[1]
Execution workflow: one fault, one observation window
1. Establish a clean baseline
Start the selected established-flow tests, fresh-connection probes and application checks before the change. Save the current roles, HA status, relevant session details, routing state and timestamps. Check both permitted and forbidden traffic while the original active peer is still forwarding.
For each probe, log at least: timestamp, test name, success/failure, latency, connection-reuse setting and error detail. Use time-synchronized systems. Keep sensitive endpoint information in your private evidence bundle, not in a public report.
2. Perform a controlled role change
Use the approved, release-specific operational procedure to suspend the current active peer or otherwise perform the documented planned switchover. Confirm which device you are connected to immediately before any disruptive action. Do not paste an unreviewed state-changing command into a production firewall.
Observe the replacement peer and the applications independently. Record the HA event, the last successful transaction before disruption, the first subsequent successful transaction and the point where sustained success resumes. Also record whether established connections reset or merely paused.
A planned role change is a useful first test, but label its scope honestly: it does not reproduce abrupt power loss, an upstream failure or isolation of an HA control path.
3. Test an approved physical or path failure separately
Only after the planned test passes, consider an independently approved uplink, monitored-path or device-loss test. Document the exact fault, the configured detection mechanism expected to react, and the restoration action. Coordinate with switching, routing and application owners.
Do not disconnect several cables at once. In particular, do not casually isolate HA control connectivity as a generic failover test: peer-isolation testing requires a platform-specific procedure and explicit safeguards against simultaneous forwarding.
4. Restore and verify before another event
Restore the failed component or peer using the approved procedure. Check intended roles, synchronization, routing, application transactions and the negative control. Account for preemption before returning a peer to service; do not assume restoring eligibility necessarily means the same final role in every deployment.
Require a stable observation window before testing another failure. If the new active peer is healthy, avoid an unnecessary immediate failback simply to restore the original dashboard appearance.
Troubleshooting matrix: role change passed, service test failed
| Symptom | Investigate first | Evidence that narrows the cause |
|---|---|---|
| New connections work; established TCP resets | State synchronization and session eligibility | HA2 health, synchronization settings, matching test session before failover |
| Only decrypted HTTPS reconnects | Documented decryption exception before blaming all HA | Compare an authorized decrypted flow with a non-decrypted control |
| Correct active role; no useful traffic | Forwarding path, adjacent switching, routing and return path | Ingress/egress observations, selected route, neighbor state and application capture |
| Outbound works; published service fails | Inbound NAT/policy and external return path | External probe, original/translated tuple and security-rule evidence |
| Traffic works; monitoring goes quiet | Per-device management configuration and log collection | Direct new-active access, collector receipt and event timestamps |
| Service returns, then drops again | Second HA event, restoration or preemption sequence | Correlated HA and application timeline |
The state-synchronization checks and decryption distinction are grounded in the vendor's documented synchronization behavior; the matrix is a proposed diagnostic order, not a claim that these are the only causes.[1][2]
For tuple-by-tuple diagnosis, use the related Palo Alto NAT troubleshooting guide. If the firewall tests pass but the application is still unavailable, continue with the F5 health-monitor troubleshooting matrix rather than repeatedly forcing firewall failovers.
Measure service recovery without false precision
Report the probe interval and timeout beside the observed interruption. A sampled probe cannot identify an exact outage boundary between observations. Likewise, a timeout can delay the recorded failure relative to the real event.
Keep three results separate: role transition, existing-flow continuity, and sustained application recovery. For example, if you choose to require several consecutive successful transactions after a failure, document that rule in advance and apply it consistently. Do not reinterpret one lucky response as stable recovery.
A useful evidence record is:
Test ID / approved fault:
Platform / PAN-OS release / management method:
Original active peer / intended final peer:
Application and flow class:
Probe interval / timeout / connection reuse:
HA event timestamp:
Last successful transaction before disruption:
First successful transaction after disruption:
Sustained recovery timestamp and rule:
Existing connection reset: yes / no / unknown
Negative policy control: pass / fail
Recovery budget: agreed value
Actual result / exception / owner:
Restoration and synchronization verified by:
Rollback and stop conditions
Stop the sequence if both peers appear to forward unexpectedly, management access is lost, the intended standby is unhealthy, required policy enforcement changes, or the application exceeds its agreed interruption budget. Treat these as reasons to execute the approved recovery plan, not to add another fault.
Define the recovery action per test: restore the isolated uplink, re-establish the monitored path, or recover the previously suspended peer only after checking the other peer's health and intended role. Never suspend both devices as a generic rollback step.
Practical takeaways
A defensible HA acceptance result is not “the passive became active.” It is a record showing which applications recovered, which established flows survived, which exceptions were expected, whether security policy stayed enforced, and whether the pair returned to a stable state.
Reuse the acceptance matrix during commissioning, after relevant design changes and during approved resilience exercises. Keep results private and sanitized when sharing lessons. For related design material, visit Start Here and the Data Center networking hub.
Post a Comment