System Design Case Studies

Designing a Distributed Cache

A distributed cache — the system behind a Redis or Memcached cluster — has to decide where each key lives across many nodes, and what happens to reads and writes when a node dies.

Consistent hashing maps both keys and nodes onto a ring, so adding or removing one node reshuffles only a small fraction of keys instead of nearly all of them
Replication factor (each key stored on N nodes) trades memory cost for read availability and durability against a single node failure
Eviction policy — LRU, LFU, or TTL-based — decides what to discard once memory fills, and should match the actual access pattern rather than being left at a default
Cache topology: client-side hashing (the client library knows the ring) versus a proxy layer (the client always talks to one endpoint, the proxy routes) — the proxy adds a hop but centralizes topology changes
The hot key problem: one extremely popular key overwhelms the single node that owns it no matter how well the rest of the ring is balanced
Cache warm-up after a restart or a cold cluster start is a real operational concern — an empty cache sends every request to the database at once, which can cascade into an outage under load

Consistent hashing ring

Client-side hashing vs a proxy layer

Eviction policies
Policy	Optimizes for	Failure mode
LRU	recency of access	a burst of one-time reads can evict genuinely hot data
LFU	frequency of access	slow to adapt when access patterns shift
TTL-based	freshness	does not account for popularity at all — can evict hot data on schedule

Looking up a key's owning node

TreeMap<Long, Node> ring = new TreeMap<>();   // hash -> node, populated with virtual nodes

Node nodeFor(String key) {
    long h = hash(key);
    Map.Entry<Long, Node> entry = ring.ceilingEntry(h);
    return entry != null ? entry.getValue() : ring.firstEntry().getValue();  // wrap around the ring
}