Part 5 of 6 · 2 min read
Blameless learning
Fix the system, not the person.
Blameless learning
BM describes Netflix as an organisation that learns from failure and adopts a blameless culture when mistakes happen. In a blameless review, the question is why the system allowed the mistake, not who made it. If people fear punishment, they hide facts, and the postmortem loses its value. A person who typed a wrong command points to a weak safeguard: why was it possible, and why did nothing catch it?
BM also says each developer at Netflix manages their own production services, which fits UDS's view that people who build a service should operate it.
Practising failure. BM lists ways to prepare: - Game days: failure is simulated by switching systems off, and teams react. Google ran them often. - Disaster tests: Google's annual DiRT exercises simulate large disasters such as earthquakes. - Chaos programs: Netflix runs programs that cause failures in production every day. Its Chaos Monkey turns off random machines in certain hours. Chaos Gorilla removes a whole availability zone. Latency Monkey simulates a slow network.
BM's message is the mindset: things will fail, and a distributed system is more exposed to failure, not less. Not everyone needs to go as far as Google or Netflix, but everyone should prepare. (Level 8 covers the designs that make a drill survivable.)
Common mistake
Ending a postmortem with "human error" as the root cause. It stops the inquiry exactly where it should start: what let one slip cause an outage?
Interview angle
Answer in your head first, then open it.
An alert fires at 3 a.m. What do you do?
Check the dashboard and runbook, mitigate first (roll back or scale out), tell the shared channel, then investigate.
What is in a good postmortem?
Impact, timeline, root cause with contributing factors, and repair items with owners and dates.
What does blameless mean?
The review asks why the system allowed the failure, not who caused it.
How do you know you can handle a failure?
Practise: game days and controlled failure in production.
Check yourself: 1. Why does blame make postmortems worse? 2. What is a game day? 3. Why is "human error" a poor stopping point?
Words you will meet
| Word | Meaning |
|---|---|
| Incident | An event that harms users beyond the SLO |
| Mitigation | A quick action that reduces harm |
| Root cause | The underlying fault that led to the incident |
| Incident lead | The person who coordinates the response |
| Postmortem | A written review with cause and repair items |
| Repair item | A task that prevents a repeat, with owner and date |
| Blameless culture | Asking why the system failed, not who did |
| Game day | A planned drill in which a failure is simulated |