Active Health Checks

Backends die. Processes crash, VMs get preempted, networks partition, disks fill. A balancer that keeps a dead backend in rotation turns one machine's failure into an error for 1 in N users, which is the opposite of its job. In this lesson your balancer starts probing its backends, reading what comes back, and keeping the pool honest.

What a probe looks like on the wire

An active health check is a synthetic request the balancer sends to every backend on a timer. The convention is a bare GET /healthz over a fresh connection:

probe sent:     GET /healthz HTTP/1.1\r\nHost: 10.0.0.11\r\nConnection: close\r\n\r\n
answer 1:       HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok
answer 2:       HTTP/1.1 503 Service Unavailable\r\nRetry-After: 5\r\n\r\n
answer 3:       (connection refused)   or   (no bytes before the deadline)

The balancer only needs the first line of the answer, the status line: HTTP-version SP status-code SP reason-phrase. The version is HTTP/ plus a digit, a dot and a digit; the status code is exactly three digits; the reason phrase is optional text that means nothing to a machine (HTTP/1.1 200 with no reason is legal). Classify it like HAProxy's option httpchk does:

Read the code from its own field. A reason phrase such as 503 Backend 200 ms slow still contains the text "200" and is still a 503.

One probe is never enough

Networks blip. A single lost packet must not eject a healthy backend, and a single lucky answer must not resurrect a dying one. Production systems use streak thresholds with hysteresis:

• after 3 consecutive failures an UP backend is marked DOWN (HAProxy's fall 3),
• after 2 consecutive passes a DOWN backend is marked UP again (rise 2).

Envoy's unhealthy_threshold and healthy_threshold and Kubernetes' failureThreshold are the same two knobs. The asymmetry is deliberate: be slow to condemn, quick to forgive.

Streaks, not totals

The word consecutive is where implementations go wrong. Each backend carries two counters, and every result clears the opposite one:

backend a, state UP, fail_streak 0, pass_streak 0

probe     verdict   fail_streak   pass_streak   state
 1        FAIL          1             0          UP
 2        FAIL          2             0          UP
 3        PASS          0             1          UP      <- a pass breaks the failure run
 4        FAIL          1             0          UP
 5        FAIL          2             0          UP      <- four failures in five probes, still UP

A totals-based checker would have condemned this backend. Intermittent isolated failures are noise; three in a row is a signal. Tune with care: fall 1 makes every blip an outage and flaps the backend in and out of the pool, while fall 10 with a 5-second interval lets a dead server eat live traffic for most of a minute.

Common mistakes

• Two failures is not DOWN. The backend stays UP and stays in the healthy list.
• Counting totals. FAIL FAIL PASS FAIL FAIL must leave the backend UP.
• Reading 4xx as healthy. For a probe a 404 means the health route is missing, so it fails. (Next lesson, the same 404 on real traffic means something else.)
• Recovery needs two in a row. After going DOWN, PASS FAIL PASS is still DOWN; the failure restarted the count.

Your exercise

The starter reads commands and prints results. POOL <b1> <b2> ... starts every backend UP with both streaks at 0 (prints OK). PROBE <backend> <answer> hands you what came back: the rest of the line is either TIMEOUT, REFUSED, or the status line as received. You classify it, update the streaks and print PASS or FAIL. STATUS prints <backend> UP|DOWN per backend in pool order, and HEALTHY prints the UP backends comma-separated in pool order, or none.