Part 4 of 8 · 2 min read
Thread dump
In plain words: A thread dump lists each thread's call stack at one moment. For a deadlock, it must also show which lock each thread waits for and who owns it. The program was run under gdb with an alarm after 2 s. With the command handle SIGALRM stop nopass, the debugger stopped the program at that moment. The listing of the threads showed:
Id Thread Where
1 main thread (LWP 1293) in nanosleep (the monitor loop)
2 worker (LWP 1296) futex_wait on the mutex at acc+144
3 worker (LWP 1297) futex_wait on the mutex at acc+48
Each Acct structure has 48 bytes (a 40-byte mutex plus an 8-byte balance), so the byte offset divided by 48 gives the account number: acc+144 is account 3, and acc+48 is account 1. Then the owner field of each mutex was printed:
(gdb) print acc[1].m.__data.__owner -> 1296
(gdb) print acc[3].m.__data.__owner -> 1297
(gdb) print acc[0].m.__data.__owner -> 0 (free)
(gdb) print acc[2].m.__data.__owner -> 0 (free)
The table of owners and waits is now complete:
| Thread | Holds | Waits for | Owner of that lock |
|---|---|---|---|
| 1296 | account 1 | account 3 | 1297 |
| 1297 | account 3 | account 1 | 1296 |
Thread 1296 waits for a lock that 1297 owns, and 1297 waits for one that 1296 owns. If the two threads are numbered 0 and 1, the array of page 2 is waits_for = {1, 0}, and the cycle finder returns the two threads. The accounts 0 and 2 are free, so no other thread is involved. The cause is plain: one worker moved money from account 1 to account 3, and another from account 3 to account 1. The first took lock 1 and then lock 3, the second took lock 3 and then lock 1. This is the interleaving of page 1.
Thread dump
A thread dump is a listing, taken at one moment, of every thread in a program with its call stack and, for blocked threads, the resource it waits for; it is the main evidence for a hang.
Example: the listing above shows two workers sleeping in futex_wait on the locks of accounts 3 and 1.
Check yourself:
1. How was the account number found from the address acc+144?
2. What does the owner field 0 mean?
3. Which two transfers caused the cycle?