Reliability & Resilience

Circuit Breakers & Retries

A circuit breaker stops calling a dependency that's already failing — protecting both the caller and the struggling callee — while a well-designed retry with exponential backoff and jitter turns a transient blip into an invisible hiccup instead of an amplified overload.

Closed: calls flow normally, failures counted. Open: calls fail immediately without touching the dependency, for a cooldown period. Half-Open: a trial call decides whether to close again
Retrying without backoff during a real outage multiplies load on an already-struggling dependency — a self-inflicted denial of service
Exponential backoff (doubling the delay each attempt) with jitter (randomizing it) prevents every client from retrying in lockstep and re-creating the exact spike that caused the failure
Only retry idempotent operations, or ones with an idempotency key (Exactly Once And Idempotency) — retrying a bare non-idempotent write can duplicate its effect
The two compose: retry a bounded number of times, but let the circuit breaker's open state stop that retrying from continuing to hammer a dependency it already knows is down

Circuit breaker state machine

Only Half-Open lets a single trial call through to test recovery

A minimal circuit breaker

enum State { CLOSED, OPEN, HALF_OPEN }

class CircuitBreaker {
    private State state = State.CLOSED;
    private int consecutiveFailures = 0;
    private final int failureThreshold = 5;
    private Instant openedAt;
    private final Duration cooldown = Duration.ofSeconds(30);

    synchronized boolean allowCall() {
        if (state == State.OPEN && Duration.between(openedAt, Instant.now()).compareTo(cooldown) > 0) {
            state = State.HALF_OPEN;          // one trial call allowed through
        }
        return state != State.OPEN;
    }

    synchronized void onSuccess() {
        consecutiveFailures = 0;
        state = State.CLOSED;
    }

    synchronized void onFailure() {
        consecutiveFailures++;
        if (state == State.HALF_OPEN || consecutiveFailures >= failureThreshold) {
            state = State.OPEN;
            openedAt = Instant.now();
        }
    }
}

Retries: backoff with jitter

Exponential backoff with full jitter

Duration backoffWithJitter(int attempt, Duration base, Duration max) {
    long capMs = Math.min(max.toMillis(), base.toMillis() * (1L << attempt));   // exponential cap
    long jitteredMs = ThreadLocalRandom.current().nextLong(capMs + 1);          // full jitter: 0..cap
    return Duration.ofMillis(jitteredMs);
}

// attempt 0: up to  base
// attempt 1: up to  2 * base
// attempt 2: up to  4 * base   ... capped at `max`

Retry strategies
Strategy	Behavior	Risk
Immediate retry	Retry instantly, no delay	Amplifies load on an already-failing dependency
Fixed delay	Same wait every attempt	Many clients retry in sync — a "thundering herd"
Exponential backoff	Delay doubles each attempt	Still synchronized if all clients started at once
Exponential backoff + jitter	Delay is randomized within a growing cap	Spreads retries out — the standard recommendation