Rate Limiting

Beyond load balancing: prevent any one client from monopolizing resources.

Token bucket (most common):

• Each client has a bucket holding up to N tokens
• Tokens regenerate at R per second
• Each request takes 1 token
• If the bucket is empty, the request is rejected (or queued)

Leaky bucket is similar but enforces a smooth output rate (bucket drains at fixed speed; requests above the rate are queued or dropped).

Sliding window: count requests in the last N seconds. More accurate than token bucket for short windows but more memory.

Distribution: in a multi-LB setup, the rate counter must be SHARED. Options:

• Centralized (Redis with INCR + EXPIRE) — accurate, adds latency
• Local with sync (each LB tracks locally, periodically syncs) — fast, approximate
• Probabilistic (allow up to capacity*N where N is the cluster size, trust statistical averaging)

HTTP headers for rate-limited responses:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1716383472