Part 6 of 6 · 4 min read
Exercises
Easy
Easy 1: Identify the key and value (10 minutes)
For a mapper that reads order log lines and needs to count orders per customer, what should the mapper emit as the key and what as the value for each line?
Done when: you state the key and value and explain why that choice groups the right rows together at reduce time.
Hint
the key should be whatever you want one total per; the value should be the number that gets combined.
Easy 2: Count the tasks (10 minutes)
An input directory has 8 file blocks, and the job is configured with 3 reducers. How many map tasks and how many reduce tasks run?
Done when: you give both numbers and explain which number came from the input and which from configuration.
Hint
map tasks come from input partitions; reduce tasks come from configuration.
Easy 3: Where does sorting happen (10 minutes)
A colleague says they need to write code to sort the key-value pairs before the reduce step. Correct them in two sentences.
Done when: your correction states that the framework sorts automatically between map and reduce, and that this is why the reducer sees its values already grouped.
Hint
point to the shuffle stage between map and reduce as doing the sorting automatically.
Medium
Medium 1: Write a mapper and reducer (25 minutes)
Write a map function and a reduce function that compute total revenue per book from a log of (book_id, price_dollars) sale events.
Done when: your mapper emits one pair per sale and your reducer sums the values for each book_id correctly.
Hint
emit (book_id, price_dollars) in the mapper, then sum the iterator of prices in the reducer for each book_id.
Medium 2: Explain computation near the data (15 minutes)
Explain in two or three sentences why the MapReduce scheduler tries to run a map task on a machine that already holds that task's input block, rather than picking any available machine at random.
Done when: your explanation names the network cost that is avoided.
Hint
avoiding network transfer of a large input block is the cost being avoided.
Medium 3: Chain two jobs (25 minutes)
The bookstore wants, for each genre, the single best-selling book. One MapReduce job can compute total sales per book. Explain what a second job would need to do, and why one job cannot produce this result alone.
Done when: you describe the second job's map and reduce logic and explain why a second sort/group stage is required.
Hint
the second job needs to group the first job's per-book totals by genre and pick the maximum within each group, which requires its own sort/group pass.
Hard
Hard 1: Diagnose a skewed key (25 minutes)
One reducer is taking far longer than the others because a single key, representing the bookstore's single most popular book, accounts for a huge share of all records. Explain why this happens given how reduce tasks are assigned, and suggest one way to reduce the imbalance.
Done when: your answer explains the hash-based assignment causing the skew and names at least one mitigation, such as splitting a hot key into sub-keys.
Hint
a key hashing to one reducer sends all of that key's records to a single machine, so a very common key creates one overloaded reducer; splitting the key into salted sub-keys and combining afterward is one fix.
Hard 2: Decide batch versus warehouse query (20 minutes)
The bookstore's analyst wants total revenue by month for the past year, a dataset of a few million rows that already lives in the warehouse's fact table. Explain whether this calls for a MapReduce job or a plain SQL query against the warehouse, and why.
Done when: your answer picks one and justifies it using the size and structure of the data.
Hint
a dataset this size already in the warehouse's fact table is exactly what SQL aggregation on a star schema is built for.
Hard 3: Trace a failure (30 minutes)
A machine running one of twelve map tasks crashes halfway through. Describe, in terms of partitions and replication, how the job can still complete correctly without the programmer writing any failure-handling code.
Done when: your trace explains that the input block is replicated elsewhere, so the task can be rerun on another machine holding a copy, and that mapper output does not need to be recomputed by any other mapper.
Hint
the scheduler reruns the failed map task on another machine that holds a replica of the same input block, and other mappers' already-sorted output is untouched.