System Design Case Studies

Designing a Rate Limiter Service

A rate limiter has to answer "allow or reject" in microseconds, stay correct when its own logic runs on multiple nodes, and not become the very bottleneck it is meant to protect against.

Algorithm choice — token bucket, leaky bucket, fixed window, or sliding window — trades burst tolerance, memory per key, and precision differently (Rate Limiting Algorithms covers the mechanics of each)
As a standalone service, the rate limiter sits in front of, or alongside as a sidecar, the services it protects, checked on every request before it is let through
Distributed counting requires a shared store: if instances do not share state, each one enforces the limit independently and the effective limit becomes limit times instance count
Redis-based implementations lean on atomic INCR plus EXPIRE (fixed window) or a sorted set of timestamps (sliding window log) so the increment-and-check step is atomic without a separate lock
Failure mode matters: if the rate limiter or its backing store is unreachable, fail open (allow traffic, risk overload) or fail closed (reject traffic, guarantee protection) must be an explicit decision
Limits are usually applied per key — per user, per IP, per API key — so storage and lookup cost scales with the number of distinct keys tracked, not just the request rate

Rate limiter placement

A Redis-backed check

Fail-open vs fail-closed
	Fail-open	Fail-closed
On limiter/store outage	traffic passes through unchecked	traffic is rejected
Availability of the protected service	not affected by the limiter's outage	reduced — depends on the limiter being up
Risk	the thing the limiter protects against can happen	a limiter outage becomes a full outage

A Redis fixed-window check

boolean allow(String key, int limit, Duration window) {
    long count = redis.incr(key);
    if (count == 1) redis.expire(key, window);   // set TTL only on the first hit in this window
    return count <= limit;
}