Part 2 of 8 · 2 min read
Symptom
In plain words: A support engineer reports: "The transfer service is up, but nothing moves since this morning. The log has no errors." The service has four accounts and a few worker threads. Each worker picks two accounts and moves one unit from the first to the second, with this function:
void transfer(int a, int b) { /* move 1 unit from account a to account b */
pthread_mutex_lock(&acc[a].m); /* first lock: the "from" account */
pthread_mutex_lock(&acc[b].m); /* second lock: the "to" account */
acc[a].bal--; acc[b].bal++;
pthread_mutex_unlock(&acc[b].m);
pthread_mutex_unlock(&acc[a].m);
}
The first step is to look, and not to guess. Three quick checks give a first answer:
- Is it using processor time?
topshows 0.0% for the process. A loop or a livelock (page 6) would show a busy processor. So the threads are asleep. - Where are they asleep? The files
/proc/<pid>/task/*/wchanshowfutex_do_waitfor the workers (page 1, section 5). They wait on mutexes, not on the network or the disk. - Is the process alive at all? Yes: the main thread still answers, so the failure is not a crash.
Zero processor use, with workers asleep on locks, is the signature of a deadlock or a lost wake-up. There are no condition variables in this code, so the deadlock is the first suspect.
Hang
A hang is a state in which a program is still alive but has stopped producing results; its cause can be a deadlock, a livelock, a wait on a slow or dead external resource, or an endless loop, and the processor use and thread states tell which.
Example: a process at 0% processor use with all workers in futex_do_wait is a hang caused by lock waits.
Check yourself:
1. Why does a processor use of 0% rule out a livelock?
2. What does futex_do_wait in wchan tell you?
3. Why are condition variables not a suspect here?