CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 6 of 7 · 3 min read

Decision table

From symptom to cause

Think of a fault-finding chart in a car manual: "engine will not start; lights dim" leads to the battery, and "lights bright" leads to the starter. Each branch needs one observation. The analogy stops in one place: a car chart is written once, and a server's chart depends on its own baseline.

In plain words: The table joins the evidence of the two tests to a conclusion and a next step.

Evidence Conclusion Next step
id near 0, r above the number of cores, wa 0 Processor saturated Find the top process (ps --sort=-pcpu), profile it, or add cores
id high, b above 0, wa high, state D Disk-bound Check wchan, the disk (Level 8), the sync rate (Level 9)
si and so above 0, free low Memory short, swapping Find the largest RSS (page 1), or add memory (Levels 6 and 7)
id high, wa 0, low load, yet slow Not a local resource: a lock, the network or a remote service strace -f for waits (page 3), network checks
st above a few per cent in a VM The host is overcommitted Ask the host operator (Level 11)

A worked example. A job takes 40 seconds when it ran in 10 last week. vmstat 1 shows us 3, sy 20, id 40, wa 37, b 2. The processor is not saturated (id 40, r is low), and wa is high with processes blocked, so the next step is the disk and the sync pattern, not a bigger processor. A check of the process finds fsync in a loop (via strace -c, page 3), as in the syncer test.

The key lines. The table does not find the answer. It tells which question to ask next. Record the baseline on a good day so that the comparison is possible, and change one thing at a time.

Bottleneck

A bottleneck is the resource that limits the throughput or the speed of a system: when it is used up or saturated, more work or faster other parts do not make the whole faster. Removing one bottleneck often shows the next.

Example: a disk-bound job does not run faster on a quicker processor.

Interview angle

A common question is "A server is slow. How do you find out why?" A good answer uses the USE method: check the processors (vmstat: id, r), memory (free, si, so), disk (b, wa, state D) and errors (dmesg), then drill into the process with ps, strace and /proc, and compares with a baseline.

Check yourself: 1. Which evidence points to the disk, and which to the processor? 2. What would you check if everything looks idle and the job is still slow? 3. Why change one thing at a time?

Words you will meet

Word Meaning
USE method Utilisation, saturation and errors for each resource.
Run queue Processes ready or running, compared with the cores.
I/O wait Idle time while processes wait for disk I/O.
State D Uninterruptible sleep, usually waiting for I/O.
Baseline The numbers of a normal day, for comparison.
Bottleneck The resource that limits the whole system.

Quick notes

  • USE: for each resource check utilisation, saturation and errors.
  • r above the number of cores is processor saturation. b and wa point to the disk. si and so point to memory.
  • Four loops on two cores each got 49 per cent: 2 / 4 = 0.5.
  • The syncer test showed wa 32 to 33, b 1, state D in submit_bio_wait, and about 47 MB/s.
  • The load average lags and counts D processes. Read the 1-minute figure for the present.
  • Record a baseline, and change one thing at a time.