Part 5 of 8 · 3 min read
Recovery
A crash in the middle of a launch.
A launch may need several requests to the cluster scheduler (for example one per task of the job). A crash can leave some sent and some not. The new leader knows only that a launch started and did not end. Two ways make it safe:
- Idempotent operations. Every request to the outside is safe to repeat. The new leader repeats all of them.
- Lookup of state. The new leader can ask the cluster scheduler what exists for this launch, and finish the missing parts.
At least one must hold. Otherwise there will be missed or double launches.
The practical method: names built in advance. Build the name of each launched task before sending anything. Include the job name and the scheduled launch time, for example report-daily@2026-11-16T09:00. Share the name through the log in the start announcement. Now every replica knows the name. After a failover, the new leader looks up each name in the cluster scheduler. If it exists, it was launched. If not, launch it with that name. A repeat with the same name cannot create a second copy.
Why the launch time matters. Take a short job that runs every minute. The leader launches it and dies before the end announcement. A slow failover takes several minutes, and the job ends meanwhile. The new leader finds no running task with the job's name, thinks it never launched, and launches it again. With the launch time in the name, the new leader sees that a task for this scheduled time existed and finished, so it does not repeat.
A worked example (made up). Job "backup" is scheduled at 02:00. The leader writes "start backup@02:00" to the log. It sends request 1 of 3 and crashes. The new leader is elected at 02:00:08. It finds an open launch for backup@02:00. It checks the cluster scheduler: task 1 exists, tasks 2 and 3 do not. It sends 2 and 3 with the prepared names, and writes the end record. The job started 8 seconds late and ran once.
Mutual exclusion. A former leader must stop at once when it loses leadership. Otherwise two leaders might both talk to the cluster scheduler. Use a lease (a time-limited right to lead, Level 5) or a check of the term number on each call, so an old leader's requests are refused. Page 6 treats this in more depth.
If neither idempotence nor lookup is possible, the designer must choose between a risk of a skipped launch and a risk of a double launch. Following the rule of this page, choose the skip.
Open launch
An open launch is a launch whose start was recorded and whose end was not, so its real state after a crash is unknown.
Example: the log has "start backup@02:00" and no "end backup@02:00", so the new leader must check it.
Check yourself: 1. How does a prepared name stop a double launch? 2. Why include the scheduled time in the name? 3. What must the old leader do when it loses leadership?