Networking & Traffic Management

Load Balancing

Distributing traffic across a pool of servers — at layer 4 (fast, protocol-agnostic) or layer 7 (content-aware, more expensive) — using an algorithm and health checks to keep it correct as the pool changes.

L4 (transport layer): balances TCP/UDP connections without inspecting payload — very fast, protocol-agnostic, cannot route on HTTP path or headers
L7 (application layer): terminates and inspects HTTP — enables routing by path, header, or cookie, at the cost of CPU for parsing and (usually) TLS termination
Algorithms: round robin (simple, ignores load), least connections (adapts to slow backends), consistent hashing (sticky, minimal remap when the pool changes)
Active health checks (periodic pings) plus passive checks (tracking live error rates) pull unhealthy backends out of rotation automatically
A load balancer is itself a single point of failure unless deployed redundantly — an active-passive pair, or a technique like anycast routing to multiple LB instances

A load balancer in front of a backend pool

Health checks continuously re-evaluate the pool; a failing instance stops receiving new traffic without any manual intervention

Load balancing algorithms
Algorithm	Behavior	Weakness
Round robin	Cycles through servers in order	Ignores actual load or request cost
Least connections	Sends to the server with fewest active connections	Needs the LB to track live connection counts
Consistent hashing	Maps a request key to a server via a hash ring	Uneven load if keys are skewed (Cache Stampede And Hot Keys)
Weighted round robin	Round robin biased by declared server capacity	Weights need manual tuning as hardware changes

A minimal least-connections picker

class LeastConnectionsBalancer {
    private final Map<String, AtomicInteger> activeConnections = new ConcurrentHashMap<>();

    String pick(List<String> healthyServers) {
        return healthyServers.stream()
            .min(Comparator.comparingInt(s -> activeConnections.getOrDefault(s, new AtomicInteger()).get()))
            .orElseThrow();
    }

    void onConnectionOpen(String server) { activeConnections.computeIfAbsent(server, s -> new AtomicInteger()).incrementAndGet(); }
    void onConnectionClose(String server) { activeConnections.get(server).decrementAndGet(); }
}