Part 7 of 8 · 4 min read
Debugging drills
Three debugging drills
Think of a fire drill: the alarm rings, and everyone follows the same few steps without discussion. The first drill is slow and the tenth is automatic. The analogy stops in one place: a fire drill has one plan, and each of these has an order that depends on the first number you read.
In plain words: Each drill gives a symptom. Answer with the first command, what the output means, and the next step.
Drill A: "The container keeps restarting. The log says exit 137." First, decode: 137 - 128 = 9, so SIGKILL. The likely sender is the out-of-memory killer of the container's memory cgroup (Level 11, page 5, where a 20 MiB group killed the test at status 137). Next steps: read memory.limit_in_bytes and memory.failcnt of the group, and compare with the program's real peak (RSS, page 1). Raise the limit or reduce the use.
Drill B: "The service says Permission denied for a file that is world-readable." First, strace -e trace=file shows -1 EACCES on one path (page 3). Next, namei -l PATH shows a directory on the path without the search bit for the service's user (Level 10, page 7). Check the real user of the process in /proc/PID/status (page 2). Fix with the smallest change, such as chmod 701 on that directory, and test that it works and that the listing is still refused.
Drill C: "The batch job takes four times longer than last week." First, vmstat 1 5 (page 4). If id is 45, wa is 33 and b is 1, the processes wait for the disk, not the processor. Next, ps -o stat,wchan shows state D in a block function, and strace -c shows a loop of fsync calls (page 3). The fix is to batch the syncs (Level 8). If instead id were 0 and r above the number of cores, the processor would be saturated, and the next step would be to find the top process.
A worked example. Time a drill: 2 minutes to say the three steps aloud for A, B and C. If you cannot name the command at each step, return to the page that teaches it. The skill is the order: decode, find the layer, narrow with one tool, and test the fix.
The key lines. In an interview, speak the order out loud. A candidate who says "first I check ... because ... then ..." shows a method, which matters more than a memorised answer.
Runbook
A runbook is a written procedure for a known problem: the symptoms, the commands to run, how to read their output and what to do next. It lets any engineer handle the problem in the same order.
Example: the three drills above, each written as a short runbook.
Interview angle
A closing question is often "Tell me about a hard bug you found." Pick a case where you read numbers before guessing, name the tools you used (vmstat, strace, gdb, /proc), say what the evidence showed, and state what you changed and how you checked that it worked.
Check yourself: 1. What does exit status 137 mean, and which cgroup file would you read? 2. Which tool finds the failing call in drill B? 3. Which two numbers separate a disk-bound job from a processor-bound one?
Words you will meet
| Word | Meaning |
|---|---|
| Trade-off | A gain in one property that costs another. |
| Turnaround / waiting / response | Finish minus arrival / turnaround minus run time / first run minus arrival. |
| Deadlock | Threads that wait for each other forever. |
| EAT | Average memory access time, weighted by hit and miss. |
| Copy-on-write | Share until written, then copy. |
| Runbook | A written procedure for a known problem. |
Quick notes
- Answer in four parts: define, mechanism, cost, example.
- Round robin q = 4 on bursts 10, 5, 8: turnaround 20.33, waiting 12.67, response 4.0. FCFS: 16.0, 8.33, 8.33.
- Deadlock needs four conditions. Lock ordering removes circular wait.
- Flat page table, 32-bit, 4 KiB pages, 4-byte entries: 4 MiB. EAT with 98 per cent hits: 103.0 ns.
- Exit status above 128 means a signal: 137 is
SIGKILL, 139 isSIGSEGV. - Debug in order: decode, find the layer, narrow with one tool, test the fix.