Part 7 of 8 · 4 min read
When things fail
Checkpoints, replay and idempotent output make a stream job restart without wrong results.
When things fail
Think of a long copy-typing job. You note the page number you finished every few pages. If you faint, a colleague reads your last note and continues from there, possibly retyping a page or two.
Page 2 explained the guarantees in detail. This page summarises how a stream job uses them in practice, following DDIA:
- The job reads from a log (page 3) and tracks its offset.
- At intervals it takes a checkpoint: it saves its state and the offsets it has reached. Flink does this using barriers in the message stream. Spark Streaming uses microbatches, small batches of about one second. Smaller batches add overhead, and larger ones delay the results.
- After a crash, a new task loads the last checkpoint and reads the log again from that offset. Output after the checkpoint is discarded or overwritten.
- Inside the framework, the result looks as though each event was processed once. Output to the outside (a database, an email) needs atomic commit or idempotent writes, or the side effect can happen twice.
The assumptions behind replay (from the book): the replayed messages must come in the same order (a log gives this), processing must be deterministic, and no other node may update the same value at the same time. Fencing may be needed when a node that was thought dead is still running.
Checkpoint
A checkpoint is a saved copy of a job's state and its input positions, taken so that the job can restart from that point after a failure.
Example: every 10 seconds the job saves "counts so far" and "read up to offset 40,000 in each partition".
A worked example (made up). A job checkpoints every 10 seconds and processes 5,000 events per second.
| Quantity | Calculation | Result |
|---|---|---|
| Events between checkpoints | 5,000 x 10 | 50,000 |
| Most to reprocess after a crash | 50,000 events | 10 seconds of work |
| Catch-up time at 5,000 per second, if the new task also handles a backlog from 30 seconds of downtime | (10 + 30) x 5,000 = 200,000 events, with capacity of say 15,000 per second | about 13 seconds |
The numbers show a trade-off. Frequent checkpoints reduce repeated work, but cost more I/O. Rare checkpoints are cheap, but make recovery slower.
A job that is slower than the incoming stream falls behind (page 3's lag). The remedy is the same as for queues: add partitions and parallel tasks, make processing faster, or drop work that does not matter.
Read it as
A stream job is a batch job with checkpoints. Everything that is hard about it comes from the fact that the input never finishes and that time cannot be trusted.
Interview angle
Answer in your head first, then open it.
Batch versus stream?
Batch is bounded, high-throughput and easy to rerun. Stream is unbounded, low-latency and needs state and checkpoints.
Event time or processing time?
Event time for correct results. Processing time is simple but wrong when there are delays or backlogs.
How do you handle late events?
Allow some lateness, then ignore them (and count the drops) or publish corrections.
Which windows do you know?
Tumbling, hopping, sliding and session.
How does a stream job recover?
Restore the latest checkpoint, replay from the saved offset, and use atomic or idempotent output.
How do you enrich events with database data?
Keep a local copy of the table updated by change data capture.
Check yourself:
- What is the cost of checkpointing too often? Too rarely?
- Why does a restart not make a stream job's internal counters wrong?
Words you will meet
| Word | Plain meaning |
|---|---|
| Stream | An unbounded sequence of events |
| Operator, job | A piece of code that processes streams |
| Complex event processing | Searching a stream for patterns of events |
| Stream analytics | Counts and averages over windows of a stream |
| Materialized view | A stored query result kept up to date |
| Event time | When the event happened |
| Processing time | When the processor handled it |
| Straggler | An event that arrives after its window was closed |
| Window | A time interval used to group events |
| Tumbling, hopping, sliding, session window | Window shapes: fixed, overlapping, interval-based, activity-based |
| Stream-stream join | Matching events of two streams within a window |
| Stream-table join | Enriching events with a table kept up to date |
| State | Data a job remembers between events |
| Checkpoint | A saved copy of state and positions |
| Microbatching | Treating a stream as tiny batch jobs |
| Backpressure | Slowing the sender when the receiver is full |