CSE Roadmap

Specialist tier · about 2 hours · 6 parts

Part 1 of 6 · 1 min read

Why this page comes first

In this module you will

  • Explain what a distributed filesystem is, and why batch processing frameworks are built on top of one.
  • Describe the map step and the reduce step, and what each one is responsible for.
  • Explain why the sort between map and reduce is central to how MapReduce works.
  • Trace how a MapReduce job spreads across many machines, and why computation runs near the data.
  • Recognize when a single-machine query is no longer enough and a batch job is the right tool.

A warehouse answers questions with SQL against one machine, or a small cluster, using indexes built in advance. Some analytical jobs are too large for that. Scanning every log line from every server for a month is one example. So is recomputing a feature for every customer. MapReduce is the classic way to split such a job across many ordinary machines.

Remember this

MapReduce makes a programmer write only two small functions, map and reduce; the framework handles splitting the data, moving it between machines, and sorting it in between.