Messaging & Stream Processing

Message Queues

A queue decouples a producer from a consumer in time and load: the producer drops a message and moves on, and one of possibly many competing consumers picks it up whenever it is ready. The queue is the shock absorber between a bursty producer and a consumer with finite capacity.

Point-to-point delivery: each message is consumed by exactly one consumer in a competing-consumers pool, unlike pub-sub's fan-out to every subscriber
Queues absorb bursts — a producer spike is buffered instead of overwhelming the consumer, at the cost of added latency for the buffered messages
Delivery is normally at-least-once: a consumer that crashes after receiving but before acknowledging causes the broker to redeliver — consumers must be idempotent (see Exactly Once And Idempotency)
A visibility timeout hides a message from other consumers while one is processing it, so two workers do not both pick up the same job
A dead-letter queue (DLQ) catches messages that fail processing repeatedly, so one poison message cannot block the whole queue forever

Competing consumers pulling from one queue

Each message goes to exactly one consumer — adding consumers increases throughput, not fan-out

An idempotent consumer with a dead-letter path

void handle(Message msg) {
    if (alreadyProcessed(msg.id())) {
        ack(msg);                       // safe no-op — this is a redelivery
        return;
    }
    try {
        process(msg);
        markProcessed(msg.id());
        ack(msg);
    } catch (RetryableException e) {
        if (msg.deliveryCount() >= MAX_RETRIES) {
            sendToDeadLetterQueue(msg, e);
            ack(msg);                   // stop retrying — a human will look at the DLQ
        } else {
            nack(msg);                  // becomes visible again after the visibility timeout
        }
    }
}

Tracking processed message ids turns "at-least-once delivery" into effectively-once processing