Part 1 of 6 · 1 min read
Why this page comes first
In this module you will
- Explain what a distributed filesystem is, and why batch processing frameworks are built on top of one.
- Describe the map step and the reduce step, and what each one is responsible for.
- Explain why the sort between map and reduce is central to how MapReduce works.
- Trace how a MapReduce job spreads across many machines, and why computation runs near the data.
- Recognize when a single-machine query is no longer enough and a batch job is the right tool.
A warehouse answers questions with SQL against one machine, or a small cluster, using indexes built in advance. Some analytical jobs are too large for that. Scanning every log line from every server for a month is one example. So is recomputing a feature for every customer. MapReduce is the classic way to split such a job across many ordinary machines.
Remember this
MapReduce makes a programmer write only two small functions, map and reduce; the framework handles splitting the data, moving it between machines, and sorting it in between.