Circuit Breakers

When a backend is overloaded, sending it more requests makes it worse: queues fill, threads run out, timeouts pile up and each retry adds to the pile. A circuit breaker is a small per-backend state machine that notices this and says "stop trying for a while, give it room to recover". It comes from an electrical fuse box, via Netflix's Hystrix, and it now lives in every service mesh and resilience library. In this lesson your balancer starts refusing to call a backend that keeps failing, and starts carefully testing it again later.

Three states

              5 consecutive errors
   CLOSED ------------------------------> OPEN
     ^                                      |
     |                                      | 30 seconds pass
     |  3 consecutive successes             v
     +-------------------------------- HALF_OPEN
                                            |
                          any error         |
              OPEN <------------------------+  (and the 30 second timer restarts)

• CLOSED is normal. Calls go to the backend, and the breaker counts consecutive errors. Any success sets the count back to 0.
• OPEN short-circuits. A call is answered immediately by the balancer with SHORT, without touching the backend at all. That is the entire point: the sick machine receives silence.
• HALF_OPEN is the careful test. After the cool-down the breaker lets calls through again. Three successes in a row close the breaker. One error reopens it and restarts the cool-down.

Compare this with last lesson's passive check. Passive ejection only counts errors, and a backend that is out is out until something external readmits it. The breaker adds the timer and the trial, so recovery happens by itself and the first requests after a repair are the test.

What counts as an error

The breaker watches real responses, with the passive rule you already know. A call is an error when its answer is not a well-formed status line (TIMEOUT, RESET, REFUSED, garbage) or has a code from 500 to 599. A 404 is the caller's mistake and is a success as far as the breaker is concerned. The starter provides this classification.

Time is lazy

Nobody runs a timer at the 30 second mark. The breaker simply remembers when it opened (opened_at) and, whenever it is next asked (a call, or a status query), checks now - opened_at >= 30. If so it is HALF_OPEN from that moment. Time here is the integer seconds set by NOW. Checking > instead of >= is the classic off-by-one: at exactly opened_at + 30 the breaker is already half open.

A short trace for one backend:

now  call           count/half_ok   state
 0   error x4       4               CLOSED
 0   success        0               CLOSED     <- the run is broken
 0   error x5       5               OPEN       opened_at = 0
 7   call           -               OPEN       answered SHORT, nothing else changes
30   status query   0               HALF_OPEN
30   success x2     2               HALF_OPEN
30   error          -               OPEN       opened_at = 30

Common mistakes

• Counting total errors instead of consecutive ones. One success must reset the CLOSED count.
• Letting SHORT calls count. A short-circuited call never reached the backend, so it is neither a success nor an error.
• Forgetting to restart the timer when a HALF_OPEN trial fails. The breaker would reopen with the old opened_at and flip straight back to half open.
• Keeping old counts across a transition. A breaker that closes must start again from 0 errors, and one that half-opens must start again from 0 successes.

Your exercise

The starter reads the commands and supplies the error classification. POOL <b1> ... creates a breaker per backend, all CLOSED (prints OK). NOW <seconds> sets the clock (prints OK). CALL <backend> <answer> runs one call through the breaker and prints OK or ERROR (what the backend did) or SHORT. STATUS prints <backend> <state> per backend in pool order. You write the state machine.