Part 2 of 8 · 2 min read
A job that never ends
A stream processor reads events as they come and writes derived events or state.
Think of a water treatment plant. A batch job is like filling a tank, treating it and releasing it. A stream job is like a pipe that is treated while the water flows, day and night. The plant never says "done".
DDIA defines a stream as data that is incrementally made available over time, with no end. Events come from user actions, sensors, and writes to databases. Stream processing is the work done on that flow. The book gives three things you can do with a stream:
- Write the events to a database, cache or search index, where other clients can query them.
- Push the events to people, for example email alerts or a live dashboard.
- Process one or more input streams to make one or more output streams, through a pipeline of several stages.
A piece of code that does the third is called an operator or a job. It is close to a Unix process or a MapReduce job: it reads inputs without changing them and writes output by appending to a different place. Partitioning and parallelism work as in MapReduce.
Stream processing
Stream processing is computing over an unbounded sequence of events, continuously, as they arrive.
Example: a job reads a stream of card payments and, for each one, checks whether the card was used in two far-apart cities within an hour.
The key difference from a batch job is that a stream never ends. Two things follow, both from the book:
- Sorting does not make sense for a dataset with no end, so the sort-merge join of page 5 cannot be used.
- Fault tolerance must change. A batch task that has been running for minutes can restart from the beginning. A stream job that has been running for years cannot.
| Batch | Stream | |
|---|---|---|
| Input | Fixed, known size | Unbounded |
| Output | At the end of the job | Continuously |
| Delay | Minutes to days | Seconds or less |
| Restart after failure | From the beginning | From a saved state |
| Sorting everything | Possible | Not possible |
Check yourself:
- Why can a stream job not restart from the beginning after a crash?
- Which join of page 5 does not work on an endless stream, and why?