CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 6 of 7 · 3 min read

Tracing cost

The cost of looking

Think of a runner who must stop at every lamp-post and tell a referee the time. The run takes much longer, and the times are not those of a normal run. The analogy stops in one place: a referee stops a runner a few times, and ptrace stops a program twice for every system call.

In plain words: Each system call of a traced process stops it twice: once at entry and once at exit, with a switch to the tracer in each case (Level 1: context switches cost). So strace slows system-call-heavy programs a great deal. The program below calls getppid 100,000 times, a very cheap call, and times the loop.

#include <stdio.h>
#include <time.h>
#include <unistd.h>

int main(void) {
    struct timespec a, b;
    clock_gettime(CLOCK_MONOTONIC, &a);
    for (int i = 0; i < 100000; i++) getppid();          /* a cheap system call */
    clock_gettime(CLOCK_MONOTONIC, &b);
    double s = (b.tv_sec - a.tv_sec) + (b.tv_nsec - a.tv_nsec) * 1e-9;
    printf("100000 calls: %.3f s, %.0f ns per call\n", s, s / 100000 * 1e9);
    return 0;
}
$ ./loop
100000 calls: 0.010 s, 103 ns per call
$ strace -o /dev/null ./loop
100000 calls: 4.676 s, 46757 ns per call

Without tracing, one call took 103 ns. Under strace, one call took 46,757 ns, about 450 times as long (46,757 / 103 = 454), and the loop went from 0.010 s to 4.7 s. The timings in a trace therefore do not describe a normal run, and tracing a busy server in production can stop it from serving. For production use, lighter tools exist: counters (/proc, page 2), sampling profilers such as perf, and tracing with eBPF, which run the filter inside the kernel without stopping the program. They were not installed on the test machine, so they are named here and not shown.

The key lines. Use strace to answer "what does it ask for?", on a test machine or a copy of the problem, and use counters and sampling to answer "where is the time going?" on a live system.

ptrace

ptrace is the system call by which one process observes and controls another: it can stop it at each system call, read and change its registers and memory, and resume it. Debuggers and strace use it.

Example: strace uses ptrace to stop the traced program at the entry and exit of each system call.

Common mistake

Reading the timings of strace -T or -c as the cost of the call in a normal run. Tracing adds tens of microseconds to each call, so use the counts and the order of calls from a trace, and use other tools for timings.

Check yourself: 1. How many times does a traced program stop for one system call? 2. By what factor did strace slow one getppid call? 3. Why is strace risky on a busy production server?

Words you will meet

Word Meaning
strace A tool that prints the system calls of a process.
ptrace The system call that lets one process watch and control another.
errno The error number of a failed call, such as ENOENT.
-c, -f, -e Count calls, follow children, choose which calls to show.
File descriptor A small number naming an open file in a process.

Quick notes

  • strace prints each system call with arguments, return value and error name.
  • A hello program makes 34 calls: 33 for start, libraries and exit, and 1 write for the work.
  • = -1 ENOENT is a missing name, EACCES a refused permission, EMFILE too many open files.
  • -e trace=file shows file calls, -f follows children, -c counts.
  • ls /tmp made 71 calls, 6 failed, 17 were mmap.
  • strace slowed one cheap call from 103 ns to 46,757 ns, about 450 times.