CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 3 of 12 · 2 min read

A stratified sample guarantees coverage that a random one can't

Pulling a fixed number from every category is the only way to be certain a rare one shows up at all.

Think of a jury selection process that deliberately seats people from every neighborhood in a city. It does not just hope that a purely random draw happens to include someone from the smallest one.

In plain words: a slice is a subset of cases defined by an attribute such as topic or user tier, and a stratified sample is one with a fixed number of cases drawn from each slice regardless of how common that slice actually is in production -- which is what lets a small eval set still guarantee it contains every category you care about, something a purely random sample cannot promise.

A worked example. Drawing 10 cases from each of the five categories (stratified sampling) found the 4%-share category 100% of the time and the 1%-share category 100% of the time, across repeated trials. Random sampling found the 4% category 88% of the time and the 1% category only 39% of the time. A third strategy -- sampling half the set from traces flagged by a weak signal (thumbs-down or an error) -- found the 1% category 70% of the time, better than random but still not guaranteed.

The key lines. From code/mini-traces/traces.py:

# from code/mini-traces/traces.py
def sample_stratified(traces, k, rng):
    by = {}
    for t in traces:
        by.setdefault(t["cat"], []).append(t)
    per = max(1, k // len(by))
    out = []
    for c, ts in by.items():
        out += rng.sample(ts, min(per, len(ts)))
    return out

Read it as: the function groups traces by category first, then samples a fixed per-count from each group independently. This is the structural reason stratified sampling guarantees coverage. A rare category gets its own dedicated sampling step. So it can never simply fail to come up, the way it can in a single pooled random draw.

Slice, stratified sample

A slice is a subset of cases defined by an attribute, such as topic or user tier. A stratified sample is a sample with a fixed number from each slice.

Example: the injection slice going from 39% coverage (random) to 100% coverage (stratified) is the entire point of slicing stated as two numbers -- if that category matters enough to track at all, it needs a sampling method that doesn't leave its presence in the eval set to chance.