CSE Roadmap

Common tier · about 2 hours · 6 parts

Part 4 of 6 · 2 min read

The postmortem

Understand the cause, agree repair items.

A postmortem is a written review after an incident. UDS gives its goal: find the root cause and list repair items that stop a repeat. It is needed for incidents that burned a significant fraction of an SLO's error budget. For example, the error-budget idea from module 1 says that an incident using 20 percent of the budget deserves one, and one using 1 percent may not.

The SRE book adds that a good postmortem is blameless. It looks for the causes that led to the incident and does not name a person as the fault. It assumes everyone involved acted in good faith with the facts they had, so people report what happened without fear.

UDS adds the team agreement: when an SLO's budget is spent, or alerts pile up, new feature work stops until reliability is restored.

A usual layout (common practice, not from the books): - Summary: what happened, in three sentences. - Impact: who was hurt, for how long, how much budget was used. - Timeline: times of detection, actions and recovery. - Root cause and contributing factors. - What went well and what went badly. - Repair items: each with an owner and a date.

A real case, from DDIA. At GitHub, an out-of-date database follower was promoted to leader. It reused primary keys that the old leader had already handed out. Those keys were also used in a separate cache store, so some private data was shown to the wrong users. This one incident shows a pattern that postmortems often find: not a single mistake but a chain, here failover, a counter, and a second store that trusted the same keys.

Postmortem

A postmortem is a written analysis of an incident that records the cause and the repair items, so that the same failure does not recur.

Example: "Checkout was down 18 minutes after a release, using 40 percent of the monthly budget. Repair items: add a schema check to the pipeline (owner A, due in 2 weeks); add a canary stage (owner B, due in 4 weeks)."

Check yourself: 1. When does UDS say a postmortem is required? 2. Why does each repair item need an owner and a date? 3. What chain of causes did the GitHub case show?