CSE Roadmap

Core tier · about 4 hours · 7 parts

Part 5 of 7 · 3 min read

Exactly-once processing in stream systems

Frameworks can give an exactly-once effect by committing outputs, state and position together.

Think of a bank clerk who records a transfer in the ledger and moves a bookmark in the same stroke. If the clerk collapses mid-way, either both the entry and the bookmark are there, or neither. On return, there is no half-done transfer to guess about.

DDIA describes batch jobs first. If a task fails in a MapReduce job, it is restarted, and its partial output is discarded. The result looks as though every record was processed exactly once, even though some work was repeated. The book calls this exactly-once semantics, and says that effectively-once would be a more honest name, because the work may in fact happen more than once while the visible effect happens once.

A stream never finishes, so a system cannot wait until the end to publish output. DDIA describes two approaches:

  • Microbatching cuts the stream into small blocks of about one second and treats each as a small batch job.
  • Checkpointing (used by Apache Flink) saves the state of the operators to durable storage from time to time. After a crash, the operator restarts from the last checkpoint and discards output produced after it.

These work inside the framework. But the book warns that as soon as output leaves the framework (a database write, a message to another broker, an email), the framework cannot undo a failed batch. A restart then makes the side effect happen twice.

To get the appearance of exactly-once, all effects of processing an event must happen if and only if the processing succeeds. The book lists them: messages sent downstream, database writes, changes to operator state, and the acknowledgment of input, which includes moving the consumer offset forward. They must all happen atomically, or none at all. This is the atomic commit problem from Level 5, module 9. The book says some systems do it efficiently by keeping everything inside one framework, so they avoid the cost of a general protocol across different technologies, and they spread the overhead by handling several messages in one transaction.

The alternative the book gives is idempotence: make each output safe to repeat. This needs a few assumptions. A restarted task must replay the same messages in the same order (a log-based broker does that), processing must be deterministic, and no other node may update the same value at the same time. When work fails over to another node, fencing may be needed to stop a node that was thought dead but is alive.

A worked example. A stream job counts page views per minute and writes the count to a database.

Approach Crash after writing 120 views but before saving the position After restart
Plain at-least-once, "add 120" The 120 events are read again The count is 240, wrong
Idempotent, "set count for minute 10:03 to 120, offset 5,000" The same events are read again The count is set to 120 again, correct
Atomic, write count and offset together The write and offset were one commit Nothing was half-saved, correct

Effectively-once processing

Effectively-once (or exactly-once) processing means a message may be handled more than once by the machinery, but its visible effect on the system's state happens a single time.

Example: a worker is restarted and re-reads message 77. It sees that offset 77 is already stored, skips the write, and the database shows one change.

Check yourself:

  • Why is microbatching alone not enough when the job sends an email?
  • Name two ways to turn at-least-once delivery into an effectively-once result.