Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-gate.
Done when: you report false-failure and detection rates for the two gates at 100 cases and three runs.
Easy 2: Use the script (30 minutes)
Write two small JSON files and run python gate.py base.json cand.json.
Done when: you see exit code 0 for a copy and 1 for a degraded candidate.
Hint
the script reads {case_id: pass_rate} JSON.
Easy 3: Sign test by hand (30 minutes)
A candidate wins 3 cases and loses 11.
Done when: you compute the exact two-sided p-value and compare with binom_two_sided(3, 14).
Hint
the smaller count is 3, n is 14; double the lower tail.
Medium
Medium 1: Slice gates (3 hours)
Add a per-slice gate with a correction for testing several slices.
Done when: you show the false-failure rate stays near 5 percent with five slices.
Hint
use a Bonferroni or Holm correction.
Medium 2: Concentrated regressions (3 hours)
Make the drop affect only 10 percent of cases, by a lot.
Done when: you compare detection by the paired gate and by a mean-difference gate.
Hint
apply the drop to a random subset of cases.
Medium 3: Tune alpha and the minimum drop (2 hours)
Sweep alpha and the minimum drop.
Done when: you give the pair that keeps false failures under 5 percent with the best power at 100 cases and three runs.
Hint
the false-failure rate is simulate(n, t, 0, "paired").
Hard
Hard 1: Real gate (8 hours)
Wire gate.py into a real repository with a real agent and 50 cases. (I did not run this, so your numbers are the ground truth.)
Done when: you report false failures over 20 unchanged runs and detection of a deliberate regression.
Hint
freeze the model version and use recorded tool outputs.
Hard 2: Baseline management (5 hours)
Store baselines with metadata and plot the trend. (I did not run this.)
Done when: a slow 1-point-per-month decay is visible even though each step passes the gate.
Hint
compare each commit to the previous baseline and to the first one.
Hard 3: Sequential testing (6 hours)
Run the suite in batches and stop early when the evidence is clear. (I did not run this.)
Done when: you show the cost saved and the false-failure rate.
Hint
apply the test after each batch with a stricter alpha.