Active Health Checks

Backends die. Processes crash, VMs get preempted, networks partition, disks fill. A balancer that keeps a dead backend in rotation converts one machine's failure into a 1-in-N error rate for every user — the exact opposite of its job. Health checking is how the pool stays honest.

Active health checks are the probing kind: the LB itself sends a synthetic request (GET /health is the convention) to every backend every few seconds and classifies the result. One probe is never enough, though. Networks blip; a single lost packet must not eject a healthy backend, and a single lucky success must not resurrect a dying one. So production systems use streak thresholds with hysteresis:

• after N consecutive failures, mark the backend DOWN (we use N = 3),
• after M consecutive successes, mark it UP again (we use M = 2).

This is exactly HAProxy's fall 3 rise 2 (its defaults), Envoy's unhealthy_threshold / healthy_threshold, and the failureThreshold on a Kubernetes liveness probe. The asymmetry is deliberate: you want to be a bit slow to condemn (tolerate blips) and reasonably quick to forgive (restore capacity), and the two knobs let you tune each independently.

Streaks, not totals

The word consecutive is where implementations go wrong. Each backend carries two counters:

Trace the sequence FAIL FAIL OK FAIL FAIL — five reports, four failures. A totals-based checker declares the backend dead; the streak-based one says UP, because the OK in the middle reset the run to zero and no run ever reached 3. That is the correct call: intermittent isolated failures are noise, three-in-a-row is a signal.

Threshold tuning is a real trade-off. fall 1 turns every transient into an outage ("flapping": the backend oscillates in and out of the pool, and each re-entry dumps a burst of traffic on it). fall 10 at a 5-second probe interval means a dead server eats live traffic for close to a minute.

Probe design matters too: a /health that just returns 200 proves the process is up, not that it can do work. Deeper checks (touch the DB, check disk) catch more but can cascade — if the database blips, every backend fails its probe simultaneously and the LB marks the entire pool down. Most shops probe shallow and let passive checks (next lesson) catch the deep failures.

Your exercise

Implement report() in the starter's HealthTracker with thresholds 3 fails / 2 successes. Commands: POOL <b1> <b2> ... (all start UP, streaks 0), REPORT <backend> OK|FAIL (update streaks and state, print OK), STATUS (print <backend> UP|DOWN per backend in pool order — not sorted), HEALTHY (comma-separated UP backends in pool order, or none).

The grader's traps:

• Two fails is not down. After REPORT a FAIL twice, STATUS must still say a UP and HEALTHY must print a.
• Streak reset. FAIL FAIL OK FAIL FAIL must leave a UP — if you count totals you'll print a DOWN and fail the hidden test.
• Recovery needs two. After going DOWN, OK OK brings it back: FAIL FAIL FAIL OK OK ends at a UP. And a FAIL between the two OKs restarts the recovery count.