CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 2 of 8 · 2 min read

Symptom

In plain words: A support engineer reports: "The transfer service is up, but nothing moves since this morning. The log has no errors." The service has four accounts and a few worker threads. Each worker picks two accounts and moves one unit from the first to the second, with this function:

void transfer(int a, int b) {              /* move 1 unit from account a to account b */
    pthread_mutex_lock(&acc[a].m);         /* first lock: the "from" account */
    pthread_mutex_lock(&acc[b].m);         /* second lock: the "to" account  */
    acc[a].bal--;  acc[b].bal++;
    pthread_mutex_unlock(&acc[b].m);
    pthread_mutex_unlock(&acc[a].m);
}

The first step is to look, and not to guess. Three quick checks give a first answer:

  1. Is it using processor time? top shows 0.0% for the process. A loop or a livelock (page 6) would show a busy processor. So the threads are asleep.
  2. Where are they asleep? The files /proc/<pid>/task/*/wchan show futex_do_wait for the workers (page 1, section 5). They wait on mutexes, not on the network or the disk.
  3. Is the process alive at all? Yes: the main thread still answers, so the failure is not a crash.

Zero processor use, with workers asleep on locks, is the signature of a deadlock or a lost wake-up. There are no condition variables in this code, so the deadlock is the first suspect.

Hang

A hang is a state in which a program is still alive but has stopped producing results; its cause can be a deadlock, a livelock, a wait on a slow or dead external resource, or an endless loop, and the processor use and thread states tell which.

Example: a process at 0% processor use with all workers in futex_do_wait is a hang caused by lock waits.

Check yourself: 1. Why does a processor use of 0% rule out a livelock? 2. What does futex_do_wait in wchan tell you? 3. Why are condition variables not a suspect here?