Part 3 of 12 · 2 min read
A stratified sample guarantees coverage that a random one can't
Pulling a fixed number from every category is the only way to be certain a rare one shows up at all.
Think of a jury selection process that deliberately seats people from every neighborhood in a city. It does not just hope that a purely random draw happens to include someone from the smallest one.
In plain words: a slice is a subset of cases defined by an attribute such as topic or user tier, and a stratified sample is one with a fixed number of cases drawn from each slice regardless of how common that slice actually is in production -- which is what lets a small eval set still guarantee it contains every category you care about, something a purely random sample cannot promise.
A worked example. Drawing 10 cases from each of the five categories (stratified sampling) found the 4%-share category 100% of the time and the 1%-share category 100% of the time, across repeated trials. Random sampling found the 4% category 88% of the time and the 1% category only 39% of the time. A third strategy -- sampling half the set from traces flagged by a weak signal (thumbs-down or an error) -- found the 1% category 70% of the time, better than random but still not guaranteed.
The key lines. From code/mini-traces/traces.py:
# from code/mini-traces/traces.py
def sample_stratified(traces, k, rng):
by = {}
for t in traces:
by.setdefault(t["cat"], []).append(t)
per = max(1, k // len(by))
out = []
for c, ts in by.items():
out += rng.sample(ts, min(per, len(ts)))
return out
Read it as: the function groups traces by category first, then samples a fixed per-count from each group independently. This is the structural reason stratified sampling guarantees coverage. A rare category gets its own dedicated sampling step. So it can never simply fail to come up, the way it can in a single pooled random draw.
Slice, stratified sample
A slice is a subset of cases defined by an attribute, such as topic or user tier. A stratified sample is a sample with a fixed number from each slice.
Example: the injection slice going from 39% coverage (random) to 100% coverage (stratified) is the entire point of slicing stated as two numbers -- if that category matters enough to track at all, it needs a sampling method that doesn't leave its presence in the eval set to chance.