Part 4 of 12 · 2 min read
Head sampling decides blind and early; tail sampling decides informed and late
Deciding whether to keep a trace before it even starts means the decision can never depend on what actually happened in it.
Think of a security guard who decides who to search by rolling dice at the door. Compare that with one who watches everyone shop and only stops the people who actually tried to walk out with something. The second one costs more attention. But the second one never lets a real thief through only because the dice came up wrong.
In plain words: head sampling decides at the start, before anything about the run is known, whether to record a trace at all, while tail sampling decides after the trace completes, using what actually happened -- an error, an unusually long duration -- which lets it deliberately keep every interesting trace while still discarding most of the boring, ordinary ones.
A worked example. Across 20,000 simulated runs, 3.9% contained a tool error. Head sampling at a 1% keep-rate stores only about 1% of those error traces too, because the decision is made before the error even happens -- it has no way to favor them. A tail rule that keeps every trace with an error, every trace over three times the median duration, plus a small 1% random base rate, stores only 5.4% of all traces overall, yet keeps 100% of the error traces and 100% of the slow ones -- the two kinds of trace you'd actually want when something breaks.
The key lines. From code/mini-tracing/tracing.py:
# from code/mini-tracing/tracing.py
def tail_sample(traces, base=0.01, slow_factor=3.0, seed=3):
"""Keep every trace with an error or a duration above slow_factor times the median, plus a small random base rate."""
rng = random.Random(seed)
med = sorted(t.duration for t in traces)[len(traces) // 2]
return [t for t in traces if has_error(t) or t.duration > slow_factor * med or rng.random() < base]
Read it as: the function needs every trace's full duration and error status computed before it can decide anything. This is the structural cost tail sampling pays for its accuracy. A head-sampling decision needs nothing but a coin flip at the start. A tail-sampling decision needs the whole trace buffered first. That is why it needs its own collector component in a real system.
Head sampling, tail sampling
Head sampling decides at the start whether to record a trace. Tail sampling decides after the trace completes, based on what happened.
Example: 1% random head sampling keeping only 1% of error traces, against a tail rule keeping 100% of them at 5.4% total storage, is the entire argument for tail sampling in two numbers -- the "what you can afford to store" question and the "what you actually need to debug" question turn out to have very different answers.