Messaging & Stream Processing
Batch vs Stream
Batch processing runs over a bounded, already-collected dataset on a schedule — high throughput, high latency. Stream processing runs continuously as data arrives — low latency, at the cost of the harder correctness problems that come with unbounded, out-of-order input.
- Batch: high throughput per unit of compute, simple reasoning (the input is fixed and known), but results are only as fresh as the last run
- Stream: results update within seconds, but every hard problem — late data, exactly-once semantics, backpressure — has to be solved instead of assumed away
- Lambda architecture runs both: a batch layer for accuracy and a speed layer for low-latency approximate results, merged at query time — at the cost of maintaining the same logic twice
- Kappa architecture drops the batch layer entirely: everything is a stream, and "batch" reprocessing is just replaying the stream from an earlier offset
- The choice is really about whether your correctness-critical logic can tolerate being expressed once (Kappa) or genuinely needs the simpler batch model for some views