CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 2 of 8 · 2 min read

Scope

What does a scheduler promise?

Think of a caretaker who must unlock a room at 9:00 each day. If the caretaker is ill, someone else must do it. And two caretakers must never unlock the same door twice in a way that causes harm.

What cron does. Unix cron runs commands at set times. A list of lines (the crontab) says when: "every hour on the hour", "at noon each day", or more complex rules. One program, crond, reads the list and launches the jobs. After a restart, only the list needs to be remembered. Launches are "fire and forget", and a simple cron does not track them.

Why one machine is not enough. The failure domain is one machine. If it is down, nothing runs. If the scheduler launches jobs on another machine (for example by SSH), there are two failure domains. With 1,000 machines, losing one machine could stop the whole scheduler.

Questions and assumptions here.

  1. Which jobs? Periodic jobs of many teams, from clean-ups to payroll runs.
  2. How many? Hundreds of thousands of job definitions, with the finest interval of 1 minute (made up).
  3. What happens on a failure? Some jobs may skip one run. Others (billing, email) must run exactly once.
  4. Who sees the state? Job owners can see whether each launch started, finished or failed.

Requirements.

  • R1. Launch each job at its scheduled times.
  • R2. Survive the loss of a machine, with a pause below one minute.
  • R3. Prefer a skipped launch over a double launch.
  • R4. Track each launch from start to finish.
  • R5. Spread load, so that many jobs do not start at once.
  • R6. Few dependencies, since many things depend on the scheduler.

Skip versus double. Jobs differ. A garbage-collection job is idempotent (Level 1): running it twice is harmless, and one skipped run is also harmless. A newsletter or a payroll job is not idempotent: a double run is a disaster that may be impossible to undo. A skipped run can be noticed by monitoring and repeated by hand. So the book's rule is: when in doubt, favour skipping over double launching.

Cron job

A cron job is a task that a scheduler starts at times set by a rule, such as every hour or each day at noon.

Example: a job "0 12 * * *" removes expired files every day at 12:00.

Check yourself: 1. Why is one machine a risk? 2. Which job type can run twice safely? 3. Why favour a skipped launch?