CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 3 of 8 · 2 min read

Reproducing

In plain words: A deadlock that cannot be repeated cannot be fixed with confidence. The aim is a reproducer: a small program or input that triggers the fault, so that it can be tried again after the fix.

The service was reduced to the transfer function, four accounts and a few threads that pick random pairs. A watchdog thread was added. It asks for the second lock with pthread_mutex_timedlock and a limit of 1 s, and prints "stuck" if the limit expires (page 1, section 5). Five runs gave:

STUCK after 9 transfers      STUCK after 125 transfers    STUCK after 196 transfers
STUCK after 388 transfers    STUCK after 703 transfers

Every run froze, after between 9 and 703 transfers. This confirms three things. The fault is in the lock logic and not a rare outside event. It needs two or more workers. And it comes quickly under load, which is why the service froze "this morning" and not after a year. With only 2 workers, three more runs also froze, after 1, 264 and 573 transfers.

If a freeze takes hours to appear, make the window bigger. Insert a short pause (50 microseconds) between the first and second lock, as in page 1. This does not add the bug. It makes the existing bug show up sooner. The pause is removed before the fix is judged.

Reproducer

A reproducer is a small program, input or script that makes a fault appear reliably and quickly, so that a fix can be tested by running it again.

Example: four accounts, random transfers, and a 1 s timeout on the second lock froze in all five runs within 703 transfers.

Check yourself: 1. Why does a pause between the two locks make the bug easier to see without adding a new bug? 2. What does the watchdog do when the timeout expires? 3. What does it tell you that all five runs froze?