CSE Roadmap

Common tier · about 2 hours · 6 parts

Part 5 of 6 · 2 min read

Blameless learning

Fix the system, not the person.

Blameless learning

BM describes Netflix as an organisation that learns from failure and adopts a blameless culture when mistakes happen. In a blameless review, the question is why the system allowed the mistake, not who made it. If people fear punishment, they hide facts, and the postmortem loses its value. A person who typed a wrong command points to a weak safeguard: why was it possible, and why did nothing catch it?

BM also says each developer at Netflix manages their own production services, which fits UDS's view that people who build a service should operate it.

Practising failure. BM lists ways to prepare: - Game days: failure is simulated by switching systems off, and teams react. Google ran them often. - Disaster tests: Google's annual DiRT exercises simulate large disasters such as earthquakes. - Chaos programs: Netflix runs programs that cause failures in production every day. Its Chaos Monkey turns off random machines in certain hours. Chaos Gorilla removes a whole availability zone. Latency Monkey simulates a slow network.

BM's message is the mindset: things will fail, and a distributed system is more exposed to failure, not less. Not everyone needs to go as far as Google or Netflix, but everyone should prepare. (Level 8 covers the designs that make a drill survivable.)

Common mistake

Ending a postmortem with "human error" as the root cause. It stops the inquiry exactly where it should start: what let one slip cause an outage?

Interview angle

Answer in your head first, then open it.

An alert fires at 3 a.m. What do you do?

Check the dashboard and runbook, mitigate first (roll back or scale out), tell the shared channel, then investigate.

What is in a good postmortem?

Impact, timeline, root cause with contributing factors, and repair items with owners and dates.

What does blameless mean?

The review asks why the system allowed the failure, not who caused it.

How do you know you can handle a failure?

Practise: game days and controlled failure in production.

Check yourself: 1. Why does blame make postmortems worse? 2. What is a game day? 3. Why is "human error" a poor stopping point?

Words you will meet

Word Meaning
Incident An event that harms users beyond the SLO
Mitigation A quick action that reduces harm
Root cause The underlying fault that led to the incident
Incident lead The person who coordinates the response
Postmortem A written review with cause and repair items
Repair item A task that prevents a repeat, with owner and date
Blameless culture Asking why the system failed, not who did
Game day A planned drill in which a failure is simulated