Part 8 of 8 · 3 min read
Exercises
Easy
Easy 1: Event time or processing time (20 minutes)
Decide which time to use for each: monthly sales per day; load on the processor right now; clicks per hour from a mobile app that uploads when online; time-out of a connection by the server.
Done when: four answers with a reason.
Hint
If the question is about when something happened, use event time.
Easy 2: Place events in windows (20 minutes)
Events at 12:00:20, 12:00:55, 12:01:10, 12:03:40. Give counts for 1-minute tumbling windows and for a session window with a 2-minute gap.
Done when: counts per window for both types.
Hint
Tumbling windows round down to the minute. A session ends after the gap.
Easy 3: Which join (20 minutes)
Name the join type: add the user's country to each click; match each ad impression to a later click; keep a timeline cache from tweets and follows; match a payment with a refund within 24 hours.
Done when: four answers.
Hint
The table that changes slowly is joined with a stream by enrichment.
Medium
Medium 1: Offset estimate (30 minutes)
A device logs an event at its clock 08:00:00, sends it at its clock 08:10:00, and the server receives it at 08:25:00 on the server's clock. Estimate the real event time. State two assumptions.
Done when: the estimate with arithmetic and two assumptions.
Hint
Offset is receive time minus send time, then add it to the event time.
Medium 2: Hopping counts (30 minutes)
A 10-minute window hops every 2 minutes. How many windows contain one event? A stream gives 1,200 events per minute. How many event-window pairs does the job count per minute?
Done when: two numbers with working.
Hint
The window length divided by the hop gives the number of windows per event.
Medium 3: Late data policy (30 minutes)
A stream of ride events has 2 percent of events arriving more than 5 minutes late. Compare three policies: close each window after 5 minutes and drop late ones; wait 30 minutes; publish corrections. State the delay, accuracy and cost of each.
Done when: a table of three policies with delay, accuracy, cost and a recommendation.
Hint
Waiting longer improves accuracy and increases delay.
Hard
Hard 1: Click-through rate pipeline (45 minutes)
Design a stream job that computes click-through rate per search result from search events and click events. State keys, window length, state kept, how to emit "no click", how to handle clicks before searches and the state size for 5,000 searches per second with a 1-hour window (50 bytes per entry).
Done when: a design and a state-size calculation.
Hint
State is searches per hour times entry size: 18,000,000 entries.
Hard 2: Recovery plan (45 minutes)
A job counts payments per merchant per minute and writes to a database. It checkpoints every 20 seconds. Describe what is read again after a crash, how to make the database write safe to repeat, and what happens if two copies of the job run at once after a false failure detection.
Done when: a recovery walk-through, a write rule and a fencing idea.
Hint
Use the window start and merchant as the key, and set the count rather than adding.
Hard 3: Enrichment at scale (50 minutes)
A stream of 100,000 events per second must be enriched with a user table of 200 million rows (100 bytes each). Compare a remote lookup per event, a local full copy per task, and a partitioned local copy. Give memory, network and staleness for each, and choose one.
Done when: a comparison table with numbers and a reasoned choice.
Hint
200 million rows at 100 bytes is about 20 GB.