Part 6 of 7 · 3 min read
Decision table
From symptom to cause
Think of a fault-finding chart in a car manual: "engine will not start; lights dim" leads to the battery, and "lights bright" leads to the starter. Each branch needs one observation. The analogy stops in one place: a car chart is written once, and a server's chart depends on its own baseline.
In plain words: The table joins the evidence of the two tests to a conclusion and a next step.
| Evidence | Conclusion | Next step |
|---|---|---|
id near 0, r above the number of cores, wa 0 |
Processor saturated | Find the top process (ps --sort=-pcpu), profile it, or add cores |
id high, b above 0, wa high, state D |
Disk-bound | Check wchan, the disk (Level 8), the sync rate (Level 9) |
si and so above 0, free low |
Memory short, swapping | Find the largest RSS (page 1), or add memory (Levels 6 and 7) |
id high, wa 0, low load, yet slow |
Not a local resource: a lock, the network or a remote service | strace -f for waits (page 3), network checks |
st above a few per cent in a VM |
The host is overcommitted | Ask the host operator (Level 11) |
A worked example. A job takes 40 seconds when it ran in 10 last week. vmstat 1 shows us 3, sy 20, id 40, wa 37, b 2. The processor is not saturated (id 40, r is low), and wa is high with processes blocked, so the next step is the disk and the sync pattern, not a bigger processor. A check of the process finds fsync in a loop (via strace -c, page 3), as in the syncer test.
The key lines. The table does not find the answer. It tells which question to ask next. Record the baseline on a good day so that the comparison is possible, and change one thing at a time.
Bottleneck
A bottleneck is the resource that limits the throughput or the speed of a system: when it is used up or saturated, more work or faster other parts do not make the whole faster. Removing one bottleneck often shows the next.
Example: a disk-bound job does not run faster on a quicker processor.
Interview angle
A common question is "A server is slow. How do you find out why?" A good answer uses the USE method: check the processors (vmstat: id, r), memory (free, si, so), disk (b, wa, state D) and errors (dmesg), then drill into the process with ps, strace and /proc, and compares with a baseline.
Check yourself: 1. Which evidence points to the disk, and which to the processor? 2. What would you check if everything looks idle and the job is still slow? 3. Why change one thing at a time?
Words you will meet
| Word | Meaning |
|---|---|
| USE method | Utilisation, saturation and errors for each resource. |
| Run queue | Processes ready or running, compared with the cores. |
| I/O wait | Idle time while processes wait for disk I/O. |
State D |
Uninterruptible sleep, usually waiting for I/O. |
| Baseline | The numbers of a normal day, for comparison. |
| Bottleneck | The resource that limits the whole system. |
Quick notes
- USE: for each resource check utilisation, saturation and errors.
rabove the number of cores is processor saturation.bandwapoint to the disk.siandsopoint to memory.- Four loops on two cores each got 49 per cent: 2 / 4 = 0.5.
- The syncer test showed
wa32 to 33,b1, stateDinsubmit_bio_wait, and about 47 MB/s. - The load average lags and counts
Dprocesses. Read the 1-minute figure for the present. - Record a baseline, and change one thing at a time.