CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 6 of 8 · 3 min read

Operations

Keeping it small, smooth and visible.

Log growth. A consensus log is a list of changes that grows without limit. Compact it with snapshots: a copy of the whole state at one moment. After a snapshot, the older log entries can be discarded. (If the log says "add 1" a thousand times, the snapshot says "the count is 1,000".) A snapshot is the most critical state: lose all snapshots and the service starts from nothing. Losing recent logs loses only changes since the last snapshot.

Where to keep them.

  • Logs: on the local disk of each replica. Three replicas means three copies. They are not written to a distributed file system, because frequent small writes are slow there. The loss of all three at once is unlikely, and it costs only the changes since the last snapshot.
  • Snapshots: on local disk too, and backed up to a distributed file system, since they are critical.
  • A new replica can fetch the latest snapshot and the logs from a running replica, so a replacement does not need any local state. Moving a replica to another machine is then a non-event.

Thundering herd. People set daily jobs to midnight. If 30 teams each launch a job that starts thousands of workers at midnight, the data centre sees a huge spike. Remedy: let a job say "any minute is fine" with a special symbol (the book uses a question mark). The scheduler picks a value by hashing the job's configuration over the range, such as the hour 0 to 23 or the minute 0 to 59. The same job always gets the same time, and different jobs spread out. Some jobs truly need a fixed time (for example a dependency on an outside event), so spikes cannot vanish entirely.

A worked example (made up). 12,000 daily jobs choose "midnight". With a free minute inside the hour (hash of the config over 0 to 59), the launches spread across 60 minutes: 200 a minute rather than 12,000 in one minute.

Other rules.

  • Overlap: if the last run is still going when the next is due, the job's setting decides: skip, queue, or allow both.
  • Missed runs after a long outage: run once, skip, or catch up, as the job's setting says. Some old cron versions (anacron) catch up daily jobs.
  • Time zones and daylight saving: keep schedules in a named zone, and decide what happens when a local time happens twice or never.
  • Retries: a failed launch can retry a number of times with a delay. A job that fails every time goes to an alert.

Monitoring. Show owners the state of each launch: scheduled, started, running, ended, failed. Alert when a job misses its launch (by more than a set delay), when a leader election takes long, when the log lag is high, and when the log is near its size limit. Owners should also monitor the effect of their jobs, since a launched job can still fail.

Read it as

The scheduler does one thing: start the right job at the right time exactly once. Every feature on this page is about making that sentence true after a crash.

Check yourself: 1. What does a snapshot save? 2. How does hashing the job's configuration help? 3. Where are snapshots backed up, and why?