CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 2 min read

What the example found

Coverage by sampling strategy, bias and RMSE of the failure-rate estimate, and the deduplication and leak-check counts.

How to read the coverage table: each row is a sampling strategy for a 50-case set drawn from 20,000 simulated traces, repeated 2,000 times; the columns are how often the set contained at least one case from a given rare category.

Strategy Contains the 4% category Contains the 1% category
Random 88% 39%
Flagged-weighted (half flagged) 99% 70%
Stratified (10 per category) 100% 100%

What to notice: random sampling misses the rarest category (1% of traffic) more often than it finds it. A coverage of 39% means a plain random eval set would contain zero injection-category examples well over half the time. This is the single clearest argument for stratified sampling when a rare category is one you specifically care about.

How to read the bias/RMSE table: each row is a sampling strategy paired with an estimator, against a true overall failure rate of 8.8%, over 600 repeated draws of 50 cases.

Strategy and estimator Bias (points) RMSE (points)
Random, naive +0.2 3.9
Random, weighted -0.3 3.8
Stratified, naive +18.8 19.6
Stratified, weighted +0.2 4.6
Flagged, naive +21.0 21.7
Flagged, weighted +16.5 17.4

What to notice: weighting fixes the stratified estimate's bias almost completely (+18.8 to +0.2). But it barely touches the flagged estimate's bias (+21.0 to +16.5). The difference is exactly the point made in Part 3. Category weights correct for a known, fixed over-representation. They do not correct for a selection process whose bias depends on the outcome itself.

How to read the deduplication and leak-check results: a single batch of fictional test text was run through both checks.

Deduplication: 43 items (20 templated order questions, 20 case-and-punctuation variants of them, 3 distinct) became 23 at a similarity threshold of 0.5, and also 23 at 0.9. The 20 questions differing only in an order number were kept as distinct, which is the limitation to fix with normalisation. Leak check: an item present in a few-shot example was flagged.

What to notice: the threshold moving from 0.5 to 0.9 without changing the result (still 23 items) suggests the genuine near-duplicates in this batch were very close matches, not borderline ones. In this particular batch, a stricter threshold did not need to trade away recall to get the same answer.

{{svg:pipeline|Collect, scrub, dedupe, sample, label and freeze.}} {{svg:results|Coverage of the rare category and error of the estimated failure rate for three sampling strategies.}}