Part 1 of 7 · 1 min read
Why this page comes fifth
In this module you will
- Tell a service, a batch job and a stream job apart by what they take in and how they are measured.
- Read a Unix pipeline that counts the most visited pages, and say why sorting scales past memory.
- Walk through a MapReduce job: map, shuffle, reduce, and name what each step does.
- Explain a sort-merge join, a broadcast hash join and a partitioned hash join, and when each fits.
- Say why batch outputs are safe to rerun, and what dataflow engines improve over MapReduce.
Pages 1 to 4 were about messages arriving one by one. Much data work does not need that. A nightly report, a rebuilt search index or a set of recommendations can be computed over a fixed pile of data. Batch processing is the oldest and simplest way to do it, and stream processing (next page) is built from the same ideas. The book used here is "Designing Data-Intensive Applications" (DDIA), chapter 10. The numbers are made up to show the idea, unless the text names a book.