Messaging & Stream Processing
Exactly-Once & Idempotency
True exactly-once delivery across an unreliable network is provably impossible in the general case (the Two Generals Problem) — what production systems actually build is at-least-once delivery combined with idempotent processing, which behaves like exactly-once from the outside.
- At-most-once: send and forget — simplest, but messages can be silently lost
- At-least-once: retry until acknowledged — messages are never lost, but duplicates are guaranteed to happen eventually
- "Exactly-once" as marketed by brokers usually means exactly-once within that broker's pipeline (e.g. Kafka's transactional producer/consumer) — it does not make an arbitrary external side effect (an email send, a payment charge) idempotent for free
- An idempotency key is a client-generated unique id for a logical operation, checked against a dedup store before the operation runs again
- Naturally idempotent operations (
SET x = 5,UPSERT) need no extra bookkeeping; naturally non-idempotent ones (x += 5, "send an email", "charge a card") do
Response charge(ChargeRequest req) {
Optional<Response> existing = dedupStore.get(req.idempotencyKey());
if (existing.isPresent()) {
return existing.get(); // replay the prior result, do not charge again
}
Response result = paymentGateway.charge(req.amount(), req.cardToken());
dedupStore.put(req.idempotencyKey(), result, Duration.ofHours(24));
return result;
}