Part 5 of 12 · 2 min read
Size the evaluation set to the specific gain you're trying to show
A tiny evaluation set can show a huge improvement convincingly, but it can't show a small one no matter how real that small improvement is.
Think of trying to tell whether a new fertilizer makes tomato plants grow slightly taller using just three plants. A dramatic difference would be obvious. But a modest, genuinely real difference would be completely lost among ordinary plant-to-plant variation.
In plain words: how many labelled evaluation items you need depends directly on how large a gain you're trying to detect. A big jump needs relatively few items to show convincingly, while a modest, equally real improvement needs many more. And there's a practical floor below which any claimed number is too noisy to trust at all.
A worked example. Detecting a jump from 50 to 90 percent accuracy needs only about 27 paired items, since a huge gap is easy to see even in a small sample. Detecting a jump from 60 to 75 percent needs about 77 items. Detecting the smaller jump from 50 to 60 percent needs about 155 items, since a smaller gain needs noticeably more evidence to pin down. Separately, even with enough items for the detection threshold, a raw score near 0.70 measured on just 50 items carries a 95 percent interval 25.4 points wide. So the module sets a floor of 100 labelled items regardless of the gain being measured. Below that, even a correctly-detected gain is too imprecise to report with confidence.
The key lines. From code/mini-scope/scope.py:
# from code/mini-scope/scope.py
MIN_ITEMS = 100 # at 100 items a 95% interval on a score near 0.7 is about 18 points wide; below that the target is hard to claim
def items_needed(baseline, target):
"""Paired items to detect the gap (80% power, two-sided 5%); disagreement taken as max(20%, 1.5 x gap)."""
gap = target - baseline
if gap <= 0:
return None
disc = min(1.0, max(0.2, 1.5 * gap))
return ceil((Z_A * sqrt(disc) + Z_B * sqrt(disc - gap * gap)) ** 2 / gap ** 2)
Read it as: the needed item count is computed directly from gap, the planned improvement. There's no single "right" evaluation set size for a capstone. It's a function of exactly what claim you intend to make, which is why the checker computes it per-scope rather than enforcing one fixed number.
Crawl-Walk-Run
Crawl-Walk-Run is a staged increase in automation, from mandatory human involvement, through internal users, to direct interaction with external users. It is the rollout pace a scope's evidence and item count should actually support, rather than a fixed schedule decided in advance.
Example: a scope that can only support detecting a 50-to-60-point gain with 155 items shouldn't be promising a confident 90-point jump to stakeholders. Crawl-Walk-Run exists so the automation level you expose to real users tracks how much evidence you've actually gathered, not how ambitious the original pitch was.