CSE Roadmap

Specialist tier · about 2 hours · 6 parts

Part 5 of 6 · 2 min read

Running across a cluster

Running it across a cluster

Think of the whole job as a factory line split across many floors. Each floor processes one bin of raw material independently. Everything then funnels into a sorting room, before the finishing floors assemble the final product.

In plain words: A MapReduce job's input is split into partitions, usually one partition per block of the input file already stored on the distributed filesystem, and one mapper task runs per partition. The number of mapper tasks is therefore decided by how the input is already split; the number of reducer tasks is chosen separately by whoever configures the job. Every mapper writes its sorted output split by which reducer should receive each key, and the framework copies each piece to its matching reducer. A single MapReduce job can only sort its data once; a job that needs a second round of sorting or a second pass over the data chains a second MapReduce job, feeding the first job's output in as the second job's input.

A worked example. The bookstore's log file is stored as 12 blocks, so 12 mapper tasks run, each on a machine already holding its block. The job is configured with 4 reducer tasks. Each mapper hashes every key it emits into one of 4 buckets, and all 12 mappers' output for bucket 2 is routed to reducer 2, which then sees every value for every key hashed into that bucket.

The key lines. No new SQL; the shape below is a job configuration, not a query.

Input: /logs/2026/ (12 blocks on HDFS)   -> 12 map tasks
Configured reducers: 4                    -> 4 reduce tasks
Mapper output partitioned by hash(key) % 4

Partition (MapReduce), map task, reduce task

In MapReduce, a partition is one chunk of the input, typically one file block, handled by one map task. A map task is the unit of work running the map function over one partition; a reduce task is the unit of work running the reduce function over one share of the sorted keys, with the number of reduce tasks set independently from the number of map tasks.

Example: the 12 log blocks above form 12 partitions and 12 map tasks, feeding into 4 reduce tasks as configured.

Quick notes

  • Distributed filesystems like HDFS spread and replicate file blocks across many machines, letting computation run near the data it needs.
  • The map function runs once per input record and only extracts a key and a value; mappers run independently of each other.
  • The framework sorts all mapper output by key before reduce runs; this sort groups every value for a key together.
  • The reduce function runs once per distinct key, over an iterator of all values for that key, and produces the final output.
  • The number of map tasks follows the number of input partitions; the number of reduce tasks is configured separately.
  • A job needing a second sort or pass chains a second MapReduce job using the first job's output as input.