CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 7 of 8 · 2 min read

Trade-offs

What to say out loud.

Trade-offs and what to say

Choice Gain Cost
Leader plus replicated log Safe failover More replicas, wait for a majority per launch
Synchronous start and end records No double launch after a crash Each launch pays a small delay
Prepared names with launch time Repeatable launches The runner must support names and lookup
Local logs, backed-up snapshots Fast writes Possible loss of the latest changes
Few dependencies Works during partial outages Builds its own storage
Hash the time ("any minute") Even load Less control for the owner
Skip over double Safe A missed run needs a follow-up

Extension: one-off and delayed jobs. A reminder "send at 14:07 tomorrow" is a job with a single time. The same machinery works. Index jobs by next launch time, and use a time-ordered store (a sorted structure or a time wheel) for millions of such jobs.

Interview angle

Answer in your head first, then open it.

What is the hard part of a distributed cron?

A crash in the middle of a launch, which can cause a missed or double run.

How do you avoid double launches?

Synchronous start and end records through a consensus log, and idempotent launches with names that include the scheduled time.

Which do you prefer, skip or double?

Skip, since it can be noticed and repeated.

How does failover work?

The group elects a new leader within seconds. It finishes open launches by looking them up by name.

How do you spread load?

Hash "any time" jobs across the range, and limit concurrent launches.

One follow-up to try. "Two leaders appear after a network partition." A good reply: the old leader on the small side cannot get a majority for its announcements, so its launches cannot proceed. A term number or lease on each request to the cluster scheduler lets the scheduler reject the old leader's calls. When the partition heals, the old leader steps down.

Check yourself: 1. What stops an old leader from launching? 2. How would a one-off job fit this design? 3. What is the cost of synchronous records?

Words you will meet

Word Meaning
Cron A tool that starts jobs at set times
Idempotent Safe to repeat
Leader and follower The replica that acts, and the replicas that copy it
Consensus log An ordered record that a majority agrees on
Snapshot A saved copy of the whole state
Open launch A launch with a start record and no end record
Thundering herd Many tasks starting at the same moment