Graceful Drain (Connection Draining)

Every deploy, scale-down, and kernel patch removes backends from a live pool. Do it bluntly — yank the backend — and every request in flight on it dies mid-response: clients see 502s and connection resets, uploads truncate, WebSockets drop. At one deploy per day across a large fleet, "hard removal" means your own release process is your biggest source of errors.

Graceful drain splits removal into two phases with a tiny state machine:

ACTIVE  -- receives new requests
DRAIN   -- receives NO new requests; in-flight requests run to completion;
           safe to terminate once inflight == 0

Mark the backend DRAIN, stop selecting it, keep serving what it already holds, and only terminate when its in-flight count hits zero. The balancer needs exactly two pieces of per-backend state to implement this: the ACTIVE/DRAIN flag and an in-flight counter (increment on dispatch, decrement on completion — the same counter least-connections uses).

This is what Kubernetes is doing during pod termination. When a pod is deleted, two things start in parallel: the pod is removed from Service endpoints (the DRAIN step — no new traffic), and the container gets its preStop hook, then SIGTERM. The app is expected to finish in-flight work and exit; if it hasn't within terminationGracePeriodSeconds (default 30 s), SIGKILL. The infamous race — the endpoint removal propagates slower than the SIGTERM arrives, so a just-signaled pod still receives a few new requests — is why so many charts put a sleep 5 in preStop.

Cloud balancers expose the same concept as a timeout knob: AWS ALB deregistration delay (default 300 s), GCP connection draining timeout, HAProxy's drain/MAINT server states, NGINX Plus's drain parameter for session-sticky upstreams.

The knob exists because of the drain-forever problem: a WebSocket or a long poll may never finish. Production drains are therefore bounded — "wait for in-flight to reach zero, or T seconds, whichever comes first" — and accept killing the stragglers. Our model keeps the unbounded version; the REMOVABLE query tells the operator who is safe to terminate right now.

Your exercise

Implement pick() in the starter's DrainablePool (the rest of the state machine is scaffolded). Commands: POOL <b1> <b2> ... (all ACTIVE, inflight 0), DRAIN <b> / UNDRAIN <b>, PICK (round-robin among ACTIVE only, increment winner's inflight, print it — or NONE if nothing is ACTIVE), DONE <b> (decrement, floor 0), STATUS (<name> <state> <inflight> sorted by name), REMOVABLE (comma-separated DRAIN backends with inflight 0, in pool order, or none).

The round-robin semantics are the one subtle spot: keep one global counter; each PICK selects actives[counter % len(actives)] from the current ACTIVE list and increments the counter. The counter does not reset when the active set changes.

Grader traps, from the real tests:

• The mod-after-drain sequence. POOL a b c: PICK → a, PICK → b (counter now 2). DRAIN a shrinks actives to [b, c], so the next PICK is index 2 % 2 = 0 → b (not c!), then c, then b. Final STATUS: a DRAIN 1, b ACTIVE 3, c ACTIVE 1. If you reset the counter on DRAIN, or skip-then-advance differently, this test fails.
• Draining doesn't erase in-flight work. A drained backend keeps its inflight count and is only REMOVABLE when it reaches 0 via DONE: drained a with 1 in flight reports none; after DONE a, REMOVABLE prints a.
• All drained: PICK prints NONE, but REMOVABLE still prints a,b — pool order, comma-joined, no spaces.
• UNDRAIN rejoins the rotation immediately: DRAIN a, two picks (b, b), UNDRAIN a, next PICK → a (counter 2 % 2 = 0 over [a, b]).