Caching

Distributed Caching

A single cache node hits a memory and throughput ceiling; distributed caches partition keys across a cluster, using consistent hashing so the cluster can grow or shrink without remapping almost everything.

A single node has a hard ceiling on both memory (how much can be cached) and throughput (how many ops/sec it can serve) — clusters of nodes (Redis Cluster, sharded Memcached) exist to lift both ceilings
Consistent hashing (a hash ring) means adding or removing one node remaps only roughly 1/N of keys, instead of nearly all of them under simple hash(key) % N
Replication inside the cache cluster trades memory for availability — a node can be lost without the keys it owned simply vanishing
Client-side sharding (the caller hashes the key to pick a node directly) avoids a proxy hop but couples every client to the cluster topology; proxy-based sharding adds a hop but centralizes that knowledge
A cold cache — after a full cluster restart or a large-scale rebalance — can send a stampede of misses straight to the origin database (Cache Stampede And Hot Keys) unless a warmup strategy is in place

Consistent hashing: adding a node remaps few keys

Only the keys that land between Node 4's position and its predecessor on the ring move — every other key stays put

Client-side vs proxy-based sharding
	Client-side	Proxy-based
Extra hop	None	One (through the proxy)
Topology awareness	Every client needs it	Centralized in the proxy
Rebalance coordination	Every client must update simultaneously	Proxy updates once

A minimal consistent-hash ring lookup

class ConsistentHashRing {
    private final TreeMap<Long, String> ring = new TreeMap<>();

    void addNode(String node) {
        for (int i = 0; i < 100; i++) ring.put(hash(node + "#" + i), node);   // virtual nodes smooth distribution
    }

    String nodeFor(String key) {
        long h = hash(key);
        Map.Entry<Long, String> entry = ring.ceilingEntry(h);
        return entry != null ? entry.getValue() : ring.firstEntry().getValue();  // wrap around the ring
    }

    private long hash(String s) { return s.hashCode() & 0xFFFFFFFFL; }
}

Virtual nodes (multiple ring positions per physical node) keep the key distribution roughly even even with few physical nodes