Passive Health Checks

Active probes have blind spots. A backend can answer GET /health in 1 ms — the handler touches nothing — while every real request fails because the connection pool to the database is exhausted. Probes also only look every few seconds: a backend can fail thousands of requests between two probes. And the probes themselves are extra load on the very servers you're worried about.

Passive health checks close the gap by scoring the traffic you're already sending. Every real response is evidence:

• Connection refused / timeout — hard failure. The backend didn't answer at all.
• 5xx status — server-side failure, counts against the backend.
• 4xx status — does NOT count. A 404 or 401 means the client sent something the server correctly rejected; a scanner probing random URLs would otherwise "fail" your whole fleet.
• Slow responses — configurable; some systems treat "over deadline" as a failure.

One nuance worth knowing: a 503 is sometimes a backend deliberately shedding load. Ejecting it for saying "please back off" can make the overload worse — production systems make the 503-counts-as-failure decision configurable.

This is how nginx's open-source health checking actually works (max_fails / fail_timeout count real request failures), and it is the core of Envoy's outlier detection, which ejects a backend after N consecutive 5xx responses (default 5) — with a cap on what fraction of the pool may be ejected at once, so a bad deploy can't empty the whole pool.

Active + passive together

The two mechanisms complement each other, and this exercise combines them with an AND:

A backend is UP iff its active state is healthy and its passive consecutive-failure streak is below 3.

Rules, precisely:

• PROBE <b> OK|FAIL drives the active side with the same streak machine as last lesson: 3 consecutive FAILs mark active-unhealthy, 2 consecutive OKs restore it.
• OBSERVE <b> <status> drives the passive side: a status in 500–599, or 0 (our encoding for timeout/connection-refused) increments the passive streak; any 2xx–4xx resets it to zero.
• The two sides are independent: probes never touch the passive streak, observations never touch the active streak.

Note the passive side recovers instantly — one good real response resets the streak — while the active side needs two clean probes. That mirrors reality: real traffic succeeding is the ground truth, while a probe is only circumstantial evidence.

Your exercise

Implement probe() and observe() in the starter's CombinedHealth (thresholds are given as constants: 3 active fails, 2 active successes, 3 passive fails). STATUS prints <backend> UP|DOWN in pool order, using the AND rule above.

What the grader checks:

• 4xx is not failure. OBSERVE a 404, 401, 403 in a row must leave a UP. Counting 4xx is the classic bug and fails a visible test.
• Consecutive means consecutive. 500 500 200 500 500 leaves a UP — the 200 reset the streak; four total 5xx never made three in a row.
• Three 5xx = DOWN, one success = back. 500 503 500 → a DOWN; a following OBSERVE a 200 → a UP.
• Sides are independent. PROBE s1 FAIL three times downs s1 even if it never served a bad response; OBSERVE s3 500/503/500 downs s3 even though its probes never failed.