Part 3 of 7 · 2 min read
Stack canary
Think of a wax seal on a letter. A reader who finds the seal broken knows the letter was opened on the way. The analogy stops in one place: a seal shows damage after it happens, and the canary also only detects, so the program must stop when it sees the break.
In plain words: A stack canary is a random value that the compiler places between the local arrays and the return address of a function. It is set when the function starts, and checked just before the function returns. An overflow that reaches the return address must first cross the canary, so the check finds a changed value, and the program stops with an error instead of returning. The name comes from the miner's canary that showed danger. The compiler flag is -fstack-protector (some functions) or -fstack-protector-all (every function).
A worked example. The same program, compiled with -fstack-protector-all, was run with the same 48 characters:
*** stack smashing detected ***: terminated
Aborted (exit status 134)
nm lists the symbol __stack_chk_fail, which the compiler calls when the check fails, in the protected program and not in the one built with -fno-stack-protector. The short input still works with the canary, and the program returns normally. The cost is a few instructions per call and a program that stops with SIGABRT (exit status 134) instead of continuing under the attacker's control. A stopped program is a lost request, and that is better than a lost machine.
The key lines. A canary does not prevent the overflow. It prevents the use of an overflow that must cross it. An attacker who can read the canary value, or an overflow that skips it, can still win.
Stack canary
A stack canary is a secret random value that the compiler stores in a function's stack frame between local buffers and the saved return address. The function checks the value before it returns, and ends the program if it has changed.
Example: gcc -fstack-protector-all adds a check that calls __stack_chk_fail when an overflow has changed the canary.
Check yourself: 1. Where does the compiler place the canary? 2. What happens when the check fails? 3. Name one thing a canary does not prevent.