Messaging & Stream Processing

Exactly-Once & Idempotency

True exactly-once delivery across an unreliable network is provably impossible in the general case (the Two Generals Problem) — what production systems actually build is at-least-once delivery combined with idempotent processing, which behaves like exactly-once from the outside.
  • At-most-once: send and forget — simplest, but messages can be silently lost
  • At-least-once: retry until acknowledged — messages are never lost, but duplicates are guaranteed to happen eventually
  • "Exactly-once" as marketed by brokers usually means exactly-once within that broker's pipeline (e.g. Kafka's transactional producer/consumer) — it does not make an arbitrary external side effect (an email send, a payment charge) idempotent for free
  • An idempotency key is a client-generated unique id for a logical operation, checked against a dedup store before the operation runs again
  • Naturally idempotent operations (SET x = 5, UPSERT) need no extra bookkeeping; naturally non-idempotent ones (x += 5, "send an email", "charge a card") do
A retried request, deduplicated by idempotency key
The client cannot tell a lost response from a lost request, so it must retry — the server absorbs the duplicate
Idempotency key check before a non-idempotent side effect
Response charge(ChargeRequest req) {
    Optional<Response> existing = dedupStore.get(req.idempotencyKey());
    if (existing.isPresent()) {
        return existing.get();              // replay the prior result, do not charge again
    }
    Response result = paymentGateway.charge(req.amount(), req.cardToken());
    dedupStore.put(req.idempotencyKey(), result, Duration.ofHours(24));
    return result;
}
The dedup store write must happen atomically with (or just after) the side effect to close the race
Sources
  • Designing Data-Intensive ApplicationsCh. 8 — The Trouble with Distributed Systems
  • Designing Distributed Systems (2nd ed.)Ch. 6 — Distributed Work Queue Patterns