A green pool member is not a promise that a customer transaction will work. If your search is “F5 pool member up but application down,” begin by writing down exactly what the monitor tests: destination, port, request, response match and application dependency. BIG-IP can check specific returned content, but an empty receive expression can accept any retrieved content; leaving both send and receive strings empty reduces the check to a connection test.[1]
This guide provides a false-green troubleshooting matrix, a reusable readiness-monitor worksheet and a controlled failure acceptance test. It is an original design example, not a report of tests executed on a BIG-IP appliance. Documentation references cover the cited BIG-IP monitor concepts and the 15.1 HTTP-monitor configuration procedure; verify field names and behavior on your installed release before making changes.
The problem: what does “up” actually prove?
An HTTP content monitor sends a request and checks the returned resource against a receive expression.[1] The diagnostic question is therefore not “why is the dashboard lying?” but “which proposition did the dashboard test?”
For this hypothetical application, the desired proposition is: this member can accept a new application request and use the dependencies required for that request. A successful connection to a web listener is a different proposition. A static landing page is another. Neither should be selected as the readiness contract by accident.
Use three separate observations in the incident record:
- Member readiness: can this specific backend accept the selected workload?
- Virtual-service reachability: can a test client reach the intended virtual service?
- Transaction success: does a representative read-only application action complete?
These are recommended evidence categories, not three built-in BIG-IP status fields. Keeping them separate prevents a backend monitor from becoming a substitute for external service testing.
False-green troubleshooting matrix
The vendor-documented mechanisms behind this matrix are monitor destination overrides, send/receive content checking and transparent/reverse operation.[1] The investigation and change recommendations are this article’s proposed workflow.
| Symptom | Evidence to collect first | Candidate explanation | Narrow correction to test |
|---|---|---|---|
| Member stays green when the application returns errors | Exact monitor request and raw response from that member | No meaningful receive expression, or a token shared by error pages | Match a dedicated readiness response; prove failure responses do not match |
| One broken member remains green while its peers work | Effective alias address and alias port on the applied monitor | The check targets a different service or shared destination | Restore per-member checking if that is the intended contract |
| Browser fails but monitor succeeds | URI, HTTP Host value and response from both requests | Default virtual host or static page is healthy, real application is not | Probe the intended application host and readiness URI |
| Database-dependent actions fail but page checks pass | Readiness endpoint implementation and dependency behavior | Endpoint only checks the web process | Define which essential dependencies must participate |
| Member is green but login or checkout fails through the virtual service | Separate backend and virtual-service tests | Backend readiness does not cover the failing end-to-end path | Add a safe synthetic transaction; do not keep editing a healthy backend check |
| Pool becomes unhealthy after a stricter check is attached | Scope of monitor association, response mismatch and dependency status | New contract fails everywhere, or the request is malformed | Restore previous association and investigate on an isolated test member |
Check the target before changing the regex
With wildcard alias settings, the monitor instance uses the associated node/member destination; explicit alias values can force another address or port.[1] Record the effective destination, not just the monitor’s friendly name.
For example, a shared alias destination may be intentional when checking a dependency, but it is not evidence that every application member is independently healthy. Ask the owner whether the intended failure domain is a member, a common dependency or a transit path. If those answers differ, split the checks and document the resulting admission policy rather than hiding the distinction in one name.
Transparent mode tests an aliased destination through the associated resource, while reverse mode marks a resource down when its test succeeds.[1] Neither is a generic “make HTTP checking stronger” option. Keep them out of this ordinary per-member readiness design unless there is a separately reviewed reason to use them.
Readiness-monitor design worksheet
The following is a proposed contract for an imaginary application. All names are invented; there are no customer addresses or production credentials.
| Field | Example or decision to record |
|---|---|
| Workload protected | New read-only application requests |
| Endpoint | /readyz on each application member |
| HTTP host | app.example.invalid — replace for the lab |
| Healthy response | A small response containing only the agreed token NF_APP_READY |
| Unhealthy response | Must not contain the healthy token; return a clearly unsuccessful application response |
| Essential dependency | The specific dependency required to serve the protected workload |
| Optional dependency | Explicitly list services whose outage must not remove every member |
| Expected destination | Each member’s service address and port; no shared alias unless intentional |
| Match behavior | Receive expression matches the readiness token, not a generic page word |
| Reverse / Transparent | No / No for this example |
| Timing | Record actual interval, timeout, recovery behavior and business detection objective |
| Change scope | Isolated test pool or one controlled member before production rollout |
| Recovery / rollback owner | Named internally, with a verified route to restore the old association |
Keep the endpoint cheap, bounded and non-destructive. Do not turn every frequent monitor request into a complete checkout transaction. The application owner should decide how to represent dependency health without creating additional load during an outage. This is a design recommendation, not a vendor guarantee about a particular application.
HTTP send string: a reviewable example
F5’s documented HTTP/1.1 send-string pattern includes a Host header and terminating blank line, with an example that also includes Connection: close.[2] The following unexecuted configuration value adapts that pattern to the hypothetical endpoint:
GET /readyz HTTP/1.1\r\nHost: app.example.invalid\r\nConnection: close\r\n\r\n
Proposed receive expression:
NF_APP_READY
Confirm how your configuration interface represents escaped line endings before saving. Inspect the actual request on the wire in a controlled test rather than assuming a pasted string was interpreted correctly.
A substring token is only safe if the endpoint contract prevents it appearing in failure responses, reflected request data or generic error templates. Do not put the success token in an error message such as “NF_APP_READY unavailable.” Have the application owner review actual healthy and unhealthy response bodies.
Do not assume a content token also verifies the HTTP status code. In this example, testing that status and body remain consistent is an explicit acceptance requirement. If both must be enforced independently, validate a suitable monitor design on the installed release rather than copying an untested multiline regex.
For HTTPS, review the actual TLS requirements separately. An HTTP Host header is not a substitute for reviewing TLS server-name behavior, certificates and authentication. Avoid translating this HTTP example blindly to an HTTPS service.
Implementation workflow: isolate before attaching
F5 documents creating a custom monitor from an existing monitor, exposing advanced settings and reviewing its configuration.[2] Use a custom object rather than changing an object shared by unrelated applications.
- Capture the baseline. Record pool membership, effective monitor association, alias settings, existing operational state and a successful client transaction. Preserve the previous configuration and association for rollback.
- Review the endpoint contract. Obtain healthy and deliberately unhealthy sample responses from the application owner. Remove secrets from stored evidence.
- Create the candidate monitor. Use the release-specific configuration interface and explicit values from the worksheet. Review inherited settings rather than accepting them unnoticed.
- Test outside the production admission path. Prefer an isolated lab pool. Do not create an accessible, unprotected test virtual service just to simplify testing.
- Exercise failure and recovery. Use the acceptance matrix below. Record observed timestamps; the table contains expected outcomes, not measured results.
- Attach in a controlled change. Confirm remaining capacity before any member can be withdrawn. Stop if the new check would remove all members unexpectedly.
- Verify the real service. Compare per-member results with an authorized client-side synthetic transaction and application-side evidence.
F5 documents Time Until Up and Manual Resume controls in its monitor configuration procedure.[2] Record their actual values before interpreting recovery. A successful response and an immediately restored production member are not interchangeable assumptions.
Failure-injection acceptance matrix
Use an isolated lab or an approved maintenance window. Do not disconnect a shared production database to test a monitor.
| Controlled scenario | Expected result for this design | Evidence required before acceptance |
|---|---|---|
| Healthy endpoint returns agreed token | Candidate check succeeds | Raw request/response, member identity and observed status |
| Application listener is unavailable | Check fails within the agreed detection objective | Failure timestamp and observed withdrawal timestamp |
| Endpoint returns an error body without token | Check fails | Saved sanitized error response and monitor state |
| Error page accidentally contains token | Treat as a test failure even if green | Demonstrated false match; fix endpoint contract before release |
| Wrong Host value reaches a default site | Must not create false readiness | Compare default-site response with the expected application response |
| Essential dependency is unavailable on isolated test instance | Readiness check fails | Application-side evidence plus observed member outcome |
| Optional dependency is unavailable | Behavior matches agreed contract | Recorded design decision; no accidental whole-pool withdrawal |
| Member recovers | Recovery follows configured policy | First healthy response, observed eligibility and operator action if required |
| Backend passes but virtual-service transaction fails | Incident remains open | Separate virtual-service investigation, not a green-only sign-off |
Do not report a fixed withdrawal time from the configured timeout alone. For this checklist, measure failure-to-observed-state timing and separately test when new client requests stop reaching the member. Set the pass/fail objective before the test, with the application and operations teams.
Rollback trigger
Rollback if the candidate request targets the wrong service, all members fail unexpectedly, or the monitor adds unacceptable dependency load. Restore the previous monitor association through the approved configuration workflow and verify both member status and a representative transaction. Do not use forced availability as a substitute for understanding a false negative.
Practical takeaways
A useful readiness monitor states a narrow, testable promise. Check its real destination, send a request for the intended application, match a response that cannot also mean failure, and prove both withdrawal and recovery. Keep external transaction testing alongside backend checks instead of treating either as a replacement for the other.
Save the worksheet and acceptance matrix with the application’s change record. Repeat them when the readiness endpoint, dependency contract or monitor association changes—not just when someone notices a red dashboard.
Related reading
- Data Center networking: related design and troubleshooting material.
- Start Here: browse the wider networking library.
- Networking resources: reference tools and documentation.
Post a Comment