Reliability & Resilience

Bulkheads & Isolation

A bulkhead partitions resources — thread pools, connection pools, whole service instances — per dependency or tenant, so one overwhelmed or misbehaving part can't exhaust resources the rest of the system needs. Named for the watertight compartments that keep a ship afloat when one section floods.

Without isolation, one slow dependency can exhaust a shared thread pool, blocking requests to healthy dependencies too — "one slow call takes down everything"
A thread-pool-per-dependency (or per-tenant) bulkhead guarantees a misbehaving one can only ever exhaust its own pool
Bulkheads and circuit breakers compose: isolation limits the blast radius of a failure, the breaker limits its duration
Applies at multiple granularities: thread pools within a process, connection pools to a database, or entire service instances/clusters dedicated per tenant
The cost of isolation is reduced resource-sharing efficiency — a pool sized for its own worst case sits partly idle most of the time

Shared pool vs bulkheaded pools

Isolating pools per dependency contains a slowdown to only the requests that actually need the slow dependency

A thread-pool bulkhead per dependency

class Bulkheads {
    private final Map<String, ExecutorService> pools = new ConcurrentHashMap<>();

    ExecutorService poolFor(String dependency, int size) {
        return pools.computeIfAbsent(dependency,
            d -> Executors.newFixedThreadPool(size));    // isolated queue + threads per dependency
    }

    <T> Future<T> call(String dependency, int poolSize, Callable<T> work) {
        return poolFor(dependency, poolSize).submit(work);
        // A slow/stuck `work` for "dependency" can only starve THIS pool, not the others.
    }
}

Isolation granularity
Level	Isolates	Typical mechanism
Thread pool	One dependency's calls from another's, within a process	Per-dependency ExecutorService, resilience4j Bulkhead
Connection pool	One database/service's connections from another's	Separate HikariCP pools per datasource
Process/container	One tenant's workload from another's, on shared hardware	cgroups, container resource limits
Service instance/cluster	A noisy tenant from all others entirely	Dedicated deployment per large tenant