Putting It All Together

You now hold every piece. This lesson wires them into one gateway that behaves like the front door of a real API: a cheap per-IP edge window, a token bucket per user whose size depends on the plan they pay for, a health-check exemption, and 429s that name the wall that was hit.

The algorithms side by side

Where the limiter lives

• Edge (CDN, WAF): volumetric abuse, bots and floods die here, before they touch your servers or your bill. Rules are coarse: per IP, per network.
• Gateway (nginx, Envoy, Kong, a cloud API gateway): per-route, per-key policy. This is where 429 and Retry-After are usually attached.
• Application: limits only your code can express, such as "3 password resets per account per day".

Each layer catches what the outer one cannot understand. The edge does not know your users; your application should never burn CPU on a flood the edge could have absorbed. Cheap checks go first, and health checks bypass the limiter entirely: a load balancer polling /healthz must never be able to take your service out of rotation by being rate limited, and its polling must not eat anyone's budget.

Sizing layers so they agree

Outer limits should only catch what the inner limits could never allow. A plan bucket with capacity C and refill r per second can pass at most C + r * 60 requests in its first minute. For the plans in this exercise:

An edge budget of 40 per IP per minute never fires on a free or pro key working from one address, and is the only wall that stops an enterprise key. Had the edge allowed 20, the pro plan's promise would be unreachable: an inconsistency arithmetic catches before production does.

The production checklist

• Shadow mode first. Log the 429s you would have sent, enforce nothing, for a week. It is the cheapest way to learn that your biggest customer's nightly sync would have been blocked on day one.
• Soft and hard thresholds. Warn at 80% of a budget; block at 100%.
• Adaptive shedding. When p99 latency or queue depth rises, tighten limits automatically; protecting the system beats honouring nominal quotas during an incident.
• Allow and deny lists for partners and known abusers.
• Humans vs machines. On a human-facing endpoint, a CAPTCHA often beats a bare 429.
• Observability per tier. Which tier fires most, top denied clients, 429 rate over time. A sudden spike is either an attack or your own misconfiguration; you want to know which in minutes.

The client's half of the contract

A limiter only works with cooperating clients: honour Retry-After, back off exponentially with jitter (a thousand clients retrying in lockstep re-create the flood), and watch X-RateLimit-Remaining to slow down before the wall. If you ship an SDK, build that in.

Study list

Stripe's engineering post on scaling an API with rate limiters (a token bucket plus layered load shedders), GitHub's documented REST hourly quota and GraphQL point costs, and Cloudflare's post on the sliding window counter. Everything in them should now read as familiar.

Common mistakes

• Spending budget on exempt traffic. A health check that counts against the edge window will eventually lock out the address it comes from.
• Letting an upgrade inherit the old bucket. A user who moves from free to pro starts with the new plan's full bucket, not the leftovers of the old one.
• Sizing an outer limit below an inner promise, so a plan's advertised rate can never be reached.

Your exercise

Assemble the gateway from the pieces you built: exempt health checks, the per-IP edge window, then the per-user plan buckets with cost-based spending.