Passive Health Checks

Active probes have blind spots. A backend can answer GET /healthz in 1 ms, because the handler touches nothing, while every real request fails since its database pool is exhausted. Probes also look only every few seconds: a backend can fail thousands of requests between two of them. And the probes are extra load on the very machines you are worried about. In this lesson your balancer starts learning from the traffic it is already carrying.

Every response is evidence

Passive checking scores real responses instead of sending synthetic ones. Look at the status line of each answer the balancer relays:

HTTP/1.1 200 OK                       fine
HTTP/1.1 404 Not Found                fine: the client asked for something that is not there
HTTP/1.1 502 Bad Gateway              error: counts against this backend
HTTP/1.1 503 Service Unavailable      error (see the note below)
(connection reset / refused / no bytes before the deadline)     error

The rule is deliberately different from the probe rule. A probe asks for one route that must exist, so a 404 there is a fault. A 404 on user traffic is the client's mistake and says nothing about the backend; if it counted, a scanner requesting random URLs would "fail" your whole fleet. So for real traffic:

• error: a status code from 500 to 599, a reset, a refusal, a timeout, or bytes that are not a status line at all,
• fine: any other well-formed status line (1xx, 2xx, 3xx, 4xx).

One nuance: a 503 is sometimes a backend deliberately shedding load. Ejecting it for saying "please back off" can deepen an overload, so production systems make that choice configurable.

This is how nginx's max_fails counts failures on real requests, and it is the heart of Envoy's outlier detection, which ejects a host after consecutive 5xx responses (5 by default) and caps how much of the pool may be ejected at once so a bad deploy cannot empty it.

Two sources, one verdict

Active and passive checking complement each other, and this exercise joins them with an AND:

A backend is UP only if its active state is up and its passive error streak is below 3.

Each side keeps its own streaks and never touches the other's. The active side is the machine from the last lesson (3 failed probes to go down, 2 passes to come back), and it is already in the starter. The passive side is new and has one more difference: it forgives at once. A single good real response clears the error streak, because real traffic succeeding is ground truth, while a probe is only circumstantial.

backend a, active UP

observe     class    passive streak   verdict
 1  500     error         1            UP
 2  503     error         2            UP
 3  200     fine          0            UP      <- the run was broken
 4  502     error         1            UP
 5  502     error         2            UP
 6  500     error         3            DOWN    <- three in a row
 7  301     fine          0            UP      <- one good response restores it

Real balancers add an ejection timer so an ejected backend is not starved of the very traffic that would prove it recovered. That timed re-admission is the circuit breaker, the next lesson.

Common mistakes

• Counting 4xx. 404, 401 and 403 in a row must leave the backend UP.
• Totals instead of streaks. 500 500 200 500 500 never reaches three in a row.
• Mixing the sides. Failed probes must not touch the passive streak, and bad responses must not touch the probe streaks. A backend with a clean passive record is still DOWN while its active state is down.
• Letting a reset or a malformed line slip through as "not a 5xx". Only a well-formed status line outside 500 to 599 is fine.

Your exercise

The starter already has the active side: PROBE <backend> <answer> classifies and counts exactly as before. You add the passive side. OBSERVE <backend> <answer> takes what the real response looked like (a status line, or TIMEOUT, REFUSED, RESET), prints ERROR or OK, and updates the passive streak. STATUS prints <backend> UP|DOWN per backend in pool order, using the AND rule.