CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 4 of 12 · 2 min read

A set that finds rare failures can't also estimate the overall rate, unless you weight it back

Deliberately over-sampling the rare, troublesome category makes that category look like most of your traffic, which it isn't.

Think of a hospital emergency room that wants to study a rare disease. It deliberately admits ten patients with it alongside ten ordinary checkups. Counting "half the patients here have this rare disease" would be absurd, because the room's population was never meant to represent the general public.

In plain words: post-stratification (weighting) corrects an estimate by weighting each slice's result by its real share of traffic, which is exactly what you need when your sample was deliberately built to over-represent rare, failure-prone slices rather than to mirror production -- without that correction, the over-represented failures inflate the estimated overall rate far beyond the truth.

A worked example. Against a true overall failure rate of 8.8%, a stratified sample read naively (just averaging the raw outcomes) overestimates by 18.8 points -- because the rare, high-failure categories (multi-step at 45%, injection at 60%) are 10 cases out of 50 each in the stratified set, instead of their true 4% and 1% shares of traffic. Weighting each category's result by its actual traffic share before combining brings the estimate's bias down to just 0.2 points -- nearly identical to a plain random sample's bias. A set built from the flagged (thumbs-down) signal stays biased by 16.5 points even after the same category weighting, because the flagging itself depends on the outcome in a way category weights can't undo.

The key lines. From code/mini-traces/traces.py:

# from code/mini-traces/traces.py
def weighted_estimate(sample, shares=None):
    """Post-stratified estimate: weight each category by its traffic share."""
    shares = shares or {c: s for c, (s, _) in CATS.items()}
    by = {}
    for t in sample:
        by.setdefault(t["cat"], []).append(t["fail"])
    tot, used = 0.0, 0.0
    for c, fs in by.items():
        tot += shares[c] * sum(fs) / len(fs)
        used += shares[c]
    return tot / used

Read it as: each category's own failure rate within the sample (sum(fs) / len(fs)) is multiplied by that category's real traffic shares[c]. It is not multiplied by how many cases of it happen to be in the sample. This is the fix for the over-representation problem. The sample can have 20% of its cases be the rare injection category. But the formula still only lets that category contribute 1% of the final weighted answer.

Post-stratification

Post-stratification (weighting) is correcting an estimate by weighting each slice by its real share of traffic.

Example: the naive stratified estimate's +18.8 point bias collapsing to +0.2 points after weighting, while the flagged estimate's bias barely moves (+21.0 to +16.5), shows the method's one limit directly: weighting fixes over-representation by a known, fixed category, but it cannot fix bias from a sampling rule whose selection depends on the very outcome being measured.