CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 5 of 12 · 2 min read

Size the evaluation set to the specific gain you're trying to show

A tiny evaluation set can show a huge improvement convincingly, but it can't show a small one no matter how real that small improvement is.

Think of trying to tell whether a new fertilizer makes tomato plants grow slightly taller using just three plants. A dramatic difference would be obvious. But a modest, genuinely real difference would be completely lost among ordinary plant-to-plant variation.

In plain words: how many labelled evaluation items you need depends directly on how large a gain you're trying to detect. A big jump needs relatively few items to show convincingly, while a modest, equally real improvement needs many more. And there's a practical floor below which any claimed number is too noisy to trust at all.

A worked example. Detecting a jump from 50 to 90 percent accuracy needs only about 27 paired items, since a huge gap is easy to see even in a small sample. Detecting a jump from 60 to 75 percent needs about 77 items. Detecting the smaller jump from 50 to 60 percent needs about 155 items, since a smaller gain needs noticeably more evidence to pin down. Separately, even with enough items for the detection threshold, a raw score near 0.70 measured on just 50 items carries a 95 percent interval 25.4 points wide. So the module sets a floor of 100 labelled items regardless of the gain being measured. Below that, even a correctly-detected gain is too imprecise to report with confidence.

The key lines. From code/mini-scope/scope.py:

# from code/mini-scope/scope.py
MIN_ITEMS = 100   # at 100 items a 95% interval on a score near 0.7 is about 18 points wide; below that the target is hard to claim

def items_needed(baseline, target):
    """Paired items to detect the gap (80% power, two-sided 5%); disagreement taken as max(20%, 1.5 x gap)."""
    gap = target - baseline
    if gap <= 0:
        return None
    disc = min(1.0, max(0.2, 1.5 * gap))
    return ceil((Z_A * sqrt(disc) + Z_B * sqrt(disc - gap * gap)) ** 2 / gap ** 2)

Read it as: the needed item count is computed directly from gap, the planned improvement. There's no single "right" evaluation set size for a capstone. It's a function of exactly what claim you intend to make, which is why the checker computes it per-scope rather than enforcing one fixed number.

Crawl-Walk-Run

Crawl-Walk-Run is a staged increase in automation, from mandatory human involvement, through internal users, to direct interaction with external users. It is the rollout pace a scope's evidence and item count should actually support, rather than a fixed schedule decided in advance.

Example: a scope that can only support detecting a 50-to-60-point gain with 155 items shouldn't be promising a confident 90-point jump to stakeholders. Crawl-Walk-Run exists so the automation level you expose to real users tracks how much evidence you've actually gathered, not how ambitious the original pitch was.