CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 7 of 8 · 3 min read

The report

A short report

In plain words: The last duty is to write the problem down, so that others learn from it. A good report is short and has the same parts every time:

Part For this case
Symptom The service froze; 0% processor use; workers in futex_do_wait
Impact No transfers completed until a restart
Cause Two code paths took the account locks in opposite orders (circular wait)
Evidence Thread dump with owners; a reproducer that froze in five of five runs; TSan lock-order reports
Fix Lock the lower-numbered account first; ignore transfers from an account to itself
Proof Stress runs without freezes; total money unchanged; no lock-order reports
Prevention Document the lock order; run TSan in tests; add the rank checker; a timeout alarm on lock waits in production

The last row matters most. The fix repairs one function. The prevention changes the process, so that the next function with two locks is written and tested with the rule in mind.

One decision remains: what to do while the service is frozen in production. The quick measure is to restart it, and save the thread dump first. Page 5 listed recovery methods; a restart is the crudest form, since it stops every thread. A better design for a service that must not freeze is a timeout on every lock wait, a count of timeouts that raises an alarm, and a retry of the transfer after a rollback (page 5, section 5).

Postmortem

A postmortem is a short written report on a failure, covering what happened, why, how it was found and fixed, and what will prevent it from happening again.

Example: the table above, with one line for each part, is a complete postmortem of the freeze.

Check yourself: 1. Which part of the report changes the future behaviour of the team? 2. Why save the thread dump before restarting the service? 3. What would a timeout on each lock wait add to the service?

Words you will meet

Word Meaning
Hang A live program that has stopped producing results
Reproducer A small test that triggers the fault reliably
Thread dump A listing of every thread's stack and what it waits for
Lock owner The thread that currently holds a lock
Lock-order inversion Two paths that take the same locks in opposite orders
Regression test A test that fails if a fixed bug returns
Postmortem A written report on a failure and its prevention
Watchdog A check that raises an alarm when something takes too long

Quick notes

  • A hang is a state in which a program is still alive but has stopped producing results; its cause can be a deadlock, a livelock, a wait on a slow.
  • A reproducer is a small program, input or script that makes a fault appear reliably and quickly, so that a fix can be tested by running it again.
  • A thread dump is a listing, taken at one moment, of every thread in a program with its call stack and, for blocked threads, the resource it waits for; it is the main evidence for a hang.
  • A lock-order inversion is a pair of code paths that take the same two locks in opposite orders; it is a possible deadlock even in a run in which no deadlock occurs.
  • A regression test is an automated test that fails if a bug that was fixed returns, so that the fix is protected against later changes.
  • A postmortem is a short written report on a failure, covering what happened, why, how it was found and fixed, and what will prevent it from happening again.