Reliability & Resilience

Fault Tolerance Patterns

Building systems that keep working — or fail predictably and cheaply — when a dependency inevitably fails. This is the umbrella discipline; timeouts, retries, circuit breakers, and bulkheads are the specific, composable tactics underneath it.

Assume every dependency will fail eventually — the network, a disk, or another service is the least reliable part of almost any architecture
A timeout is the most basic and most frequently missing mechanism: a call with none can hang a thread — and everything waiting on it — forever
Redundancy (multiple instances, replicas) only helps if failure is detected and rerouted around quickly; undetected failure is just a landmine
Isolation (Bulkheads And Isolation) stops one failing dependency from starving resources unrelated requests need
"Fail fast, fail clearly" beats "hang indefinitely" — a fast, explicit failure lets a caller retry, fall back, or degrade instead of piling up behind it

Every pattern in this domain answers the same question from a different angle: when (not if) a dependency fails, how does the rest of the system keep functioning instead of failing along with it? Treat this topic as the map; the others are the specific tools.

Matching a failure mode to a pattern

Failure mode → pattern
Failure mode	Pattern	Covered in
A call hangs instead of failing	Timeout on every network call, no exceptions	this topic
A dependency is down or erroring repeatedly	Circuit breaker stops calling it for a cooldown	Circuit Breakers And Retries
A transient blip (one bad response)	Retry with exponential backoff and jitter	Circuit Breakers And Retries
One slow dependency exhausts a shared pool	Bulkhead: isolated pool per dependency	Bulkheads And Isolation
Total load exceeds capacity	Rate limiting and load shedding	Rate Limiting Algorithms, Graceful Degradation And Load Shedding

The bug: a network call with no timeout

// No timeout configured — this call can block the calling thread indefinitely
// if the remote server accepts the connection but never responds.
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder(uri).build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

// Fixed: an explicit connect timeout AND a per-request timeout.
HttpClient safeClient = HttpClient.newBuilder()
    .connectTimeout(Duration.ofSeconds(2))
    .build();
HttpRequest safeRequest = HttpRequest.newBuilder(uri)
    .timeout(Duration.ofSeconds(3))     // caps time waiting for the response too
    .build();