Part 6 of 8 · 2 min read
Correctness
Making the totals right.
Exactly-once. A crash could count a batch twice or not at all. The aggregation service reads events from the first queue, computes the results, writes them to the second queue, and records its read position. If the write to the queue and the saving of the read position happen together as one atomic step, a restart cannot repeat or skip. The book calls this an atomic commit, a feature in queue and stream tools. A downstream writer with an idempotent key (page 4) completes the guarantee.
Recalculation. If a bug corrupted the counts, they are rebuilt from raw data. A recalculation service reads the raw records for the affected time, sends them through a separate copy of the aggregation service, so live traffic is not slowed, and writes the corrected results to the second queue. The same code serves live and replay.
Batch and stream. Stream processing handles unbounded data as it arrives and gives near real-time results. Batch processing handles a fixed set of data, such as a day of raw data. A design that has both is called a lambda architecture, and its cost is two code bases that must agree. A kappa architecture uses one stream engine for both live data and replay. This design follows kappa, since recalculation goes through the same service.
Reconciliation. At the end of each day, a batch job re-counts from raw data and compares the result with the stream results. It fixes the small gaps left by very late events. A worked example (made up). The stream says ad 7 had 1,000,000 clicks yesterday, and the batch says 1,000,230. The difference of 230, or 0.023 per cent, is written back as a correction.
Failures. If an aggregate node fails, a new node starts from the last saved position in the queue and rebuilds its in-memory counts by reading events again. Saving a snapshot of the state from time to time shortens the replay.
Read it as
The stream path is the quick tally during the day. The batch path is the careful count at closing time, and the final bill follows it.
Check yourself: 1. What makes the result exactly-once? 2. Why does recalculation use a separate copy of the service? 3. What does end-of-day reconciliation correct?