Part 7 of 8 · 2 min read
Trade-offs
What to say.
Trade-offs and what to say
| Choice | Gain | Cost |
|---|---|---|
| Queues between parts | Isolation, scaling | More parts to run |
| Event time | Accurate counts | Late events to handle |
| Longer watermark | More accuracy | More delay |
| Pre-computed filters | Fast reads | More rows per filter |
| Raw data kept | Recalculation, debugging | Storage |
| Kappa architecture | One code path | Replay is heavy |
| Hot key splitting | Even load | More merge work |
Scaling. Add partitions to the queues and nodes to the DAG, since the pipeline is partitioned by ad id. Give each stage its own resources. Watch the delay between a click and its count, and the lag of each queue (page 1). Alert if counts drop to zero for an active ad, which often means a broken pipeline and not a lack of clicks (Level 9).
Interview angle
Answer in your head first, then open it.
Why a stream pipeline?
The rate is high, and results are needed within minutes.
How do you get exactly-once?
Atomic commit of the result and the read position, plus idempotent writes.
How do you handle late events?
Event time with a watermark, and a nightly batch correction.
How do you find the top N?
A heap in each aggregate node, merged in a reduce step.
What if you find a bug in the counting?
Replay from raw data through a separate copy of the service.
One follow-up to try. "Detect fake clicks." A good reply: add a filtering step in the map stage that drops clicks from known bad IP addresses and clicks that repeat from one user too fast, using rules at first and a model later. Keep the raw data of dropped clicks for audits. Fraud filtering changes billing, so it must be logged and reproducible, and a replay should give the same answer.
Check yourself: 1. How would you notice that the pipeline is broken? 2. Why keep dropped clicks? 3. What does the watermark cost?
Words you will meet
| Word | Meaning |
|---|---|
| CTR | Clicks divided by views |
| Event time | When the click happened |
| Processing time | When the server handled it |
| Watermark | Extra time a window stays open for late events |
| Tumbling window | Fixed slices that do not overlap |
| Sliding window | A window that always covers the latest M minutes |
| Lambda architecture | Separate batch and stream paths |
| Kappa architecture | One stream path for live and replay |