Messaging & Stream Processing
Stream Processing
Computing continuously over an unbounded stream of events rather than a finite batch — aggregations, joins, and alerts that update as new events arrive, using windows to make "sum the last 5 minutes" a well-defined, finite question over an infinite stream.
- Event time (when something actually happened) vs processing time (when the system saw it) diverge under network delay and retries — correctness usually needs event time
- Windowing makes aggregation over an infinite stream tractable: tumbling (fixed, non-overlapping), sliding (overlapping), and session (gap-based) windows are the three common shapes
- Watermarks are a stream processor's way of saying "I do not expect events older than time T anymore" — they bound how long a window waits for late data before closing
- Stateful operators (a running count, a join across two streams) need their state checkpointed so a crash can resume rather than restart from zero
- Not the same thing as Java's
java.util.stream— that is a lazy, one-shot, in-process pipeline over data already fully available in memory; distributed stream processing (Flink, Kafka Streams, Spark Streaming) is a continuously running, fault-tolerant, unbounded computation
| Window | Shape | Example use |
|---|---|---|
| Tumbling | Fixed size, non-overlapping | "Requests per minute", each event in exactly one window |
| Sliding | Fixed size, overlapping | "Rolling 5-minute average", updated every 30 seconds |
| Session | Gap-based, variable size | "User activity session", closes after N minutes of inactivity |