Part 7 of 8 · 2 min read
Trade-offs
What to say out loud.
Trade-offs and what to say
| Choice | Gain | Cost |
|---|---|---|
| Leader plus replicated log | Safe failover | More replicas, wait for a majority per launch |
| Synchronous start and end records | No double launch after a crash | Each launch pays a small delay |
| Prepared names with launch time | Repeatable launches | The runner must support names and lookup |
| Local logs, backed-up snapshots | Fast writes | Possible loss of the latest changes |
| Few dependencies | Works during partial outages | Builds its own storage |
| Hash the time ("any minute") | Even load | Less control for the owner |
| Skip over double | Safe | A missed run needs a follow-up |
Extension: one-off and delayed jobs. A reminder "send at 14:07 tomorrow" is a job with a single time. The same machinery works. Index jobs by next launch time, and use a time-ordered store (a sorted structure or a time wheel) for millions of such jobs.
Interview angle
Answer in your head first, then open it.
What is the hard part of a distributed cron?
A crash in the middle of a launch, which can cause a missed or double run.
How do you avoid double launches?
Synchronous start and end records through a consensus log, and idempotent launches with names that include the scheduled time.
Which do you prefer, skip or double?
Skip, since it can be noticed and repeated.
How does failover work?
The group elects a new leader within seconds. It finishes open launches by looking them up by name.
How do you spread load?
Hash "any time" jobs across the range, and limit concurrent launches.
One follow-up to try. "Two leaders appear after a network partition." A good reply: the old leader on the small side cannot get a majority for its announcements, so its launches cannot proceed. A term number or lease on each request to the cluster scheduler lets the scheduler reject the old leader's calls. When the partition heals, the old leader steps down.
Check yourself: 1. What stops an old leader from launching? 2. How would a one-off job fit this design? 3. What is the cost of synchronous records?
Words you will meet
| Word | Meaning |
|---|---|
| Cron | A tool that starts jobs at set times |
| Idempotent | Safe to repeat |
| Leader and follower | The replica that acts, and the replicas that copy it |
| Consensus log | An ordered record that a majority agrees on |
| Snapshot | A saved copy of the whole state |
| Open launch | A launch with a start record and no end record |
| Thundering herd | Many tasks starting at the same moment |