CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 5 of 12 · 3 min read

A good evaluation report and architecture note are built like a good experiment

The two documents a reviewer trusts most are the one that shows its baseline and interval, and the one that shows what else you considered.

Think of a lab notebook versus a press release about the same discovery. The notebook shows the control group, the error bars and the failed attempts. The press release just shows the headline result.

In plain words: an evaluation report earns trust by reporting the same things a careful experiment would — the data, a baseline to compare against, results with confidence intervals, and failure examples grouped by cause. An architecture note earns trust differently, by listing each real decision with its alternatives and what was given up. Together they split what a reader needs into functional transparency (what the system does and why) and architectural transparency (how the parts actually coordinate).

A worked example. The fictional evaluation report states its baseline (a simple rules system scoring 61%, interval 53% to 69%) right next to the system's own score (81%, interval 75% to 87%). It also states the resulting gain (20 points, interval 11 to 29). A number alone would say nothing. Stated next to its baseline and interval, it says the gain is real and roughly how large. The cost ($0.0092 per ticket) and latency (3.2s median, 5.7s at the 95th percentile) sit beside it. An evaluation report without cost and latency columns is an incomplete picture of the trade-off, even if the accuracy number is impressive.

The key lines. From code/mini-portfolio/sample/eval.py:

# from code/mini-portfolio/sample/eval.py
res = {
    "n_items": n,
    "baseline_rules": {"accuracy": float(rules.mean()), "ci": boot_ci(rules.astype(float))},
    "agent": {"accuracy": float(agent.mean()), "ci": boot_ci(agent.astype(float))},
    "gain": {"points": float(100 * diff.mean()), "ci_points": [100 * v for v in boot_ci(diff, seed=1)]},
    "cost_per_ticket_usd": float(cost),
    "latency_s": {"p50": float(np.percentile(lat, 50)), "p95": float(np.percentile(lat, 95))},
}

Read it as: baseline_rules and agent are computed and stored side by side, with the same bootstrap confidence-interval function (boot_ci) applied to both. The baseline isn't an afterthought tacked onto the report. It's structurally equal to the system's own result in the data, which is what makes the comparison honest rather than decorative.

Evaluation report, architecture note, functional and architectural transparency

The evaluation report is the evidence document: data, systems compared, metrics, results with intervals, error analysis and reproduction steps. The architecture note is a short document of components, data flow, decisions with alternatives, failure modes and cost. Functional transparency tells users what the system is doing and why. Architectural transparency shows developers how the parts coordinate.

Example: the baseline-plus-interval pair above is functional transparency, aimed at a user deciding whether to trust the system's output. An architecture note entry explaining why a rules baseline was chosen for comparison, over say a different agent architecture, is architectural transparency instead, aimed at a developer deciding whether to extend it.