Part 1 of 4 · 1 min read
Overview
Agent practice moves on research results: new prompting schemes, tuning recipes, memory designs, benchmark scores. Teams that adopt a method for a one-point gain on a noisy benchmark pay engineering cost for nothing; teams that dismiss everything miss real advances. A fast, structured reading and a small reproduction on your own data decides which is which.
About the example, and what is real
Real: the structured note and checklist code, the simulations (equal-skill comparisons, seed spread, tuning budget, test-set selection, many benchmarks), the bootstrap interval, the invented Simpson's-paradox example, the sample-size formulas, and 21 tests in code/mini-papers. Assumed: item difficulty drawn from a Beta(2,2) distribution with a sigmoid skill model, run-to-run spreads of 0.5, 1 and 2 points, a configuration spread of 1.5 points, 500-item benchmarks, and 20 percent disagreement between models. Not real: the "paper" is fictional and no actual paper was read or reproduced.