Distributed Rate Limiting

Everything so far kept its counters in one process. Put ten API servers behind a load balancer and each one counts only the requests it happened to receive, so a client spread evenly across them gets ten times its budget: your "100 per minute" is quietly 1,000. After this lesson your gateway keeps its counters in a shared store, a stand-in for Redis, and keeps answering when that store goes away.

Where does the counter live?

Every distributed design answers that one question differently.

Edge networks use the middle row and accept the overshoot. This lesson builds the first.

The recipe, in bytes

A fixed window needs one counter per client per window, and Redis offers exactly that. The app server sends two commands (RESP framing; \r\n is two bytes). The first adds one and returns the new count:

*2\r\n$4\r\nINCR\r\n$17\r\nrl:alice:28333333\r\n        reply  :1\r\n

When that reply is 1, the key has just been created, so the server also tells Redis when to throw it away, the instant the window ends:

*3\r\n$8\r\nEXPIREAT\r\n$17\r\nrl:alice:28333333\r\n$10\r\n1700000040\r\n        reply  :1\r\n

The key is rl: + client + : + the window index, 28333333 = floor(1700000000 / 60). Pseudocode for the decision:

count = INCR key
if count == 1: EXPIREAT key (window end)
allowed = count <= limit

Three details carry the weight.

• The window index is part of the key. A leaked key (the server died between the two commands) cannot corrupt the next window, because the next window uses a different key. The expiry is memory hygiene, not correctness.
• INCR counts refused requests too, because the server increments before it knows the verdict. The counter keeps climbing past the limit, so X-RateLimit-Remaining must be clamped at zero.
• No server identity in the key. The point is that every server increments the same counter.

Why INCR and not GET then SET

Redis runs commands one at a time, so ten servers calling INCR at once receive ten different values. The broken alternative reads the count, adds one, and writes it back:

Two requests were admitted and the books say one. A token bucket has the same problem with two fields (tokens and last), which a single command cannot update. The fix is to move the whole read-compute-write into one server-side script: Redis runs a script start to finish without interleaving anything else. Two refinements: set an expiry inside the script so idle clients vanish, and take now from Redis, not from the app servers, whose clocks drift.

When the store is down

Decide before the incident. Fail closed (refuse everything) turns a Redis blip into a total outage of your own API. Fail open (admit everything, count nothing, alert loudly) risks overload for the length of the outage, and it is the usual choice. Either way the decision belongs in code you can test.

Common mistakes

• Counters per server. Keying by server brings back the multiplier.
• A key with no expiry. Memory grows with every client and window you have ever seen.
• Counting while Redis is unreachable, or refusing requests during the outage.
• A negative Remaining once the counter is past the limit.

Your exercise

Replace the local counter with the shared store: 10 requests per client per 60-second window across all servers, expiring every key at its window end, and answering 200 without headers while the store is down.