Putting It All Together

You have built the parts of a balancer one at a time. This lesson is where they become one program: a request arrives as raw bytes, passes through several decisions, and leaves as a routed, counted response. When it works, you have a small version of what NGINX, HAProxy, Envoy and a cloud load balancer do on every request.

The pipeline

Each request goes through the same stages, in this order:

raw HTTP request
   |
   v
1. identify the client      X-Forwarded-For -> first address, else "unknown"
   |
   v
2. rate limit               token bucket for that client           -> 429 + Retry-After
   |
   v
3. choose a backend         Cookie route=... names an UP backend?  -> sticky
   |                        otherwise round-robin over UP backends -> new + Set-Cookie
   |                        nobody UP                              -> 503
   v
4. record                   per-backend request counter

Health runs beside it. Probe results (PROBE) move backends between UP and DOWN with the same fall 3 / rise 2 hysteresis you wrote earlier, and stage 3 only ever considers backends that are currently UP.

The order is a design decision, and the tests check it:

• Rate limiting comes before backend selection. Rejecting a flood must cost as little as possible, so a limited request never reaches the rotation. It does not advance the round-robin counter and is not counted against any backend.
• The token is spent before the backend is chosen. A request that ends in a 503 because nobody is UP has still used a token. Without that, clients could hammer a failing pool for free.
• A sticky hit takes no turn. Only a new assignment advances the single round-robin counter.
• Identity comes from the proxy chain. X-Forwarded-For: 203.0.113.7, 10.0.0.2 lists the original client first and each proxy that relayed it after. Take the first entry. Requests with no such header share one client called unknown.

Reusing what you have

Nothing here is new. The rules for each stage are the ones you already implemented, and the stage descriptions repeat them so you do not have to scroll back:

What a production balancer adds on top is long: TLS termination and SNI (you built the certificate choice), HTTP/2 and gRPC per-call balancing, WebSocket upgrades, connection draining, retries with jitter, hedged requests, circuit breakers (you built one), canary percentages, anycast and DNS steering. Each is another stage in this same shape: look at the request, consult some state, decide, record.

Common mistakes

• Wrong stage order. Choosing a backend first and rate limiting after makes limited requests consume rotation turns and inflates the backend counters.
• Not spending the token on a 503.
• Shared state leaking between stages. The bucket belongs to the client, the counter to the pool, the streaks to the backend. A probe must not touch a bucket.
• Taking the last X-Forwarded-For address instead of the first, or the whole header value as the client.
• Forgetting that POOL mid-run starts a fresh rotation, health and counters for backends, but does not hand every client a fresh bucket.

Your exercise

The starter reads the commands and the raw requests. You write the stage functions and glue them in handle. METRICS prints one lb_requests_total{backend="<b>"} <n> line per pool backend sorted by name, then lb_rate_limited_total <n> and lb_unavailable_total <n>.