CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 7 of 8 · 4 min read

Debugging drills

Three debugging drills

Think of a fire drill: the alarm rings, and everyone follows the same few steps without discussion. The first drill is slow and the tenth is automatic. The analogy stops in one place: a fire drill has one plan, and each of these has an order that depends on the first number you read.

In plain words: Each drill gives a symptom. Answer with the first command, what the output means, and the next step.

Drill A: "The container keeps restarting. The log says exit 137." First, decode: 137 - 128 = 9, so SIGKILL. The likely sender is the out-of-memory killer of the container's memory cgroup (Level 11, page 5, where a 20 MiB group killed the test at status 137). Next steps: read memory.limit_in_bytes and memory.failcnt of the group, and compare with the program's real peak (RSS, page 1). Raise the limit or reduce the use.

Drill B: "The service says Permission denied for a file that is world-readable." First, strace -e trace=file shows -1 EACCES on one path (page 3). Next, namei -l PATH shows a directory on the path without the search bit for the service's user (Level 10, page 7). Check the real user of the process in /proc/PID/status (page 2). Fix with the smallest change, such as chmod 701 on that directory, and test that it works and that the listing is still refused.

Drill C: "The batch job takes four times longer than last week." First, vmstat 1 5 (page 4). If id is 45, wa is 33 and b is 1, the processes wait for the disk, not the processor. Next, ps -o stat,wchan shows state D in a block function, and strace -c shows a loop of fsync calls (page 3). The fix is to batch the syncs (Level 8). If instead id were 0 and r above the number of cores, the processor would be saturated, and the next step would be to find the top process.

A worked example. Time a drill: 2 minutes to say the three steps aloud for A, B and C. If you cannot name the command at each step, return to the page that teaches it. The skill is the order: decode, find the layer, narrow with one tool, and test the fix.

The key lines. In an interview, speak the order out loud. A candidate who says "first I check ... because ... then ..." shows a method, which matters more than a memorised answer.

Runbook

A runbook is a written procedure for a known problem: the symptoms, the commands to run, how to read their output and what to do next. It lets any engineer handle the problem in the same order.

Example: the three drills above, each written as a short runbook.

Interview angle

A closing question is often "Tell me about a hard bug you found." Pick a case where you read numbers before guessing, name the tools you used (vmstat, strace, gdb, /proc), say what the evidence showed, and state what you changed and how you checked that it worked.

Check yourself: 1. What does exit status 137 mean, and which cgroup file would you read? 2. Which tool finds the failing call in drill B? 3. Which two numbers separate a disk-bound job from a processor-bound one?

Words you will meet

Word Meaning
Trade-off A gain in one property that costs another.
Turnaround / waiting / response Finish minus arrival / turnaround minus run time / first run minus arrival.
Deadlock Threads that wait for each other forever.
EAT Average memory access time, weighted by hit and miss.
Copy-on-write Share until written, then copy.
Runbook A written procedure for a known problem.

Quick notes

  • Answer in four parts: define, mechanism, cost, example.
  • Round robin q = 4 on bursts 10, 5, 8: turnaround 20.33, waiting 12.67, response 4.0. FCFS: 16.0, 8.33, 8.33.
  • Deadlock needs four conditions. Lock ordering removes circular wait.
  • Flat page table, 32-bit, 4 KiB pages, 4-byte entries: 4 MiB. EAT with 98 per cent hits: 103.0 ns.
  • Exit status above 128 means a signal: 137 is SIGKILL, 139 is SIGSEGV.
  • Debug in order: decode, find the layer, narrow with one tool, test the fix.