Graceful Drain

Every deploy, scale-down and kernel patch takes backends out of a live pool. Do it bluntly, by yanking the machine, and every request in flight on it dies mid-response: clients see 502s and resets, uploads truncate, WebSockets drop. With one deploy a day across a large fleet, hard removal makes your own release process your biggest source of errors. In this lesson your balancer learns to take a backend out politely.

Two phases

Graceful drain splits removal into two steps with a tiny state machine:

ACTIVE ---- DRAIN b ----> DRAIN ---- REMOVE b (when removable) ----> gone
   ^                        |
   +------ UNDRAIN b -------+

ACTIVE   receives new requests
DRAIN    receives NO new requests; the requests it already holds run to completion

Marking a backend DRAIN stops it being selected while it keeps serving what it holds. It is safe to terminate once its in-flight count reaches zero. The balancer needs only two pieces of per-backend state: the ACTIVE/DRAIN flag and an in-flight counter (up on dispatch, down on completion, the same counter least-connections uses).

Drain forever is not an option

A WebSocket or a long poll may never finish. A drain with no limit can block a deploy for ever, so production drains are bounded: wait until in-flight reaches zero or a timeout passes, whichever comes first, and then kill the stragglers. That bound is a knob everywhere: AWS ALB deregistration delay (default 300 s), GCP connection draining timeout, HAProxy's MAINT and drain states, NGINX Plus's drain parameter.

Here DRAIN <backend> [timeout] records when the drain began; with a timeout it also fixes a deadline at now + timeout. A DRAIN backend is removable when its in-flight count is 0 (clean), or, if it has a deadline, once now has reached it (forced). REMOVABLE lists them in pool order, marking forced ones with a trailing !.

This is what Kubernetes does

When a pod is deleted, two things start in parallel: the pod is taken out of the Service endpoints (the drain step: no new traffic), and the container gets its preStop hook and then SIGTERM. The application should finish in-flight work and exit; if it has not within terminationGracePeriodSeconds (30 s by default), it gets SIGKILL, the deadline above. The notorious race is that endpoint removal propagates slower than SIGTERM arrives, so a just-signalled pod still receives a few new requests. That is why so many charts put a sleep 5 in preStop: it keeps the pod serving while the drain spreads.

The one subtle spot: the rotation

PICK is round-robin over the ACTIVE backends in pool order, with one global counter that is never reset when the active set changes. Each pick takes actives[counter mod count] from the current ACTIVE list, then adds one to the counter.

pool a b c, counter 0
PICK -> a          counter 1
PICK -> b          counter 2
DRAIN a            actives are now [b, c]
PICK -> index 2 mod 2 = 0 -> b     counter 3     (not c: the list shrank under the counter)
PICK -> index 3 mod 2 = 1 -> c     counter 4

That is correct behaviour for a shared counter and it is what the tests pin down. Resetting the counter on DRAIN, or advancing past the drained backend first, gives a different (also reasonable, but not the specified) sequence.

Common mistakes

• Treating DRAIN as removal. A drained backend keeps its in-flight count and keeps being listed until it is actually removed.
• Declaring a backend removable while requests are still in flight and no deadline has been reached.
• Dropping the requests silently. REMOVE reports how many in-flight requests were killed.
• Resetting the rotation counter when the active set changes.
• Letting UNDRAIN forget the in-flight count. The backend simply rejoins the rotation immediately.

Your exercise

The starter holds the state and parses commands. You write the pick, the removability rule and REMOVE. POOL, NOW, DRAIN, UNDRAIN, DONE and STATUS are described in the exercise text.