CSE Roadmap

Common tier · about 5 hours · 13 parts

Part 6 of 13 · 4 min read

What the example found

The traffic-mix break-even, the design-variant comparison, weight-uncertainty win rates, the critical weights, the scaling pitfall, and the decision-record linter.

How to read the break-even table: each row is a different assumed share of tool-heavy traffic; the value column is how much a quality point would have to be worth for the gain to justify the token cost.

Share of tool-heavy traffic Direct score Multi-agent score Gain Value per point needed (assumed $1 per million tokens)
5% 9.55 9.30 -0.26 never
10% 9.41 9.29 -0.12 never
20% 9.12 9.28 0.16 $0.094
30% 8.83 9.27 0.44 $0.034
50% 8.25 9.25 1.00 $0.015
70% 7.67 9.23 1.56 $0.010
90% 7.09 9.21 2.12 $0.007

What to notice: the required value per point drops fast as traffic mix rises. At 90% tool-heavy traffic, even a quality point worth less than a cent justifies the switch. Below the ~14% break-even, the multi-agent system never clears the bar, no matter how cheap a quality point is.

How to read the design-variant table: each row is one of five architecture candidates from module 3's simulation, scored against a fixed weighted scale.

Design variant (module 3 sim., assumed quality) p95 (s) Cost Quality Weighted score Passes p95 <= 8 s and q >= 0.72
As designed 10.4 $0.0125 0.78 0.347 no (p95)
Cap loop at 2 9.1 $0.0115 0.77 0.414 no (p95)
Small planner 7.7 $0.0063 0.75 0.515 yes
Small planner + cap 7.0 $0.0061 0.73 0.520 yes
No router (big model only) 11.3 $0.0156 0.84 0.377 no (p95)

What to notice: the two variants with the highest raw quality ("as designed" and "no router") both fail the latency requirement. They are eliminated before the weighted score even matters. The eventual top weighted score (0.520) belongs to the variant with the lowest quality among the five. That only looks surprising until you remember the constraint filter already removed its higher-quality competitors.

How to read the weight-confidence table: each row is a level of confidence (concentration) in the stated base weights; the columns are each surviving variant's probability of winning across simulated weight draws.

Weight confidence (concentration) Small planner wins Small planner + cap wins
5 45% 55%
20 40% 60%
50 35% 65%
200 23% 77%

What to notice: even at the highest confidence level tested, the losing option still wins 23% of the time across simulated weight draws. Uncertainty in the weights never fully resolves for a comparison this close. That is the table's whole point: it reports a probability, not a guarantee.

How to read the sensitivity table: each row is a different check on how fragile the winning decision is.

Sensitivity of the choice Result
With 10% noise on the raw values (confidence 20) 45% / 55%
p95 weight below 0.37 flips to the cap variant
Quality weight above 0.43 flips to the plain small planner
Cost weight never flips it
Winner's quality: assumed vs break-even 0.73 vs 0.728
All five options, weights uncertain (confidence 20) winner 60%, small planner 30%, big model only 10%, the other two 0%

What to notice: the winner's assumed quality (0.73) sits just 0.002 above the point where it would lose (0.728). A margin this thin is a direct signal to go measure the real quality number rather than keep debating the weights. The module states that as its own explicit rule.

How to read the normalisation-pitfall table: each row is a different scaling choice or option set at fixed weights 0.6/0.4.

Normalisation pitfall (weights 0.6 / 0.4) Winner
Min-max, options A and B A
Min-max, adding dominated option D B
Fixed targets, with or without D B

What to notice: adding option D flips the min-max winner from A to B simply by stretching the scale. Option D should never be chosen anyway, since it is dominated. The fixed-target scaling gives the same answer (B) whether or not D is present at all. This is the rank reversal made concrete.

How to read the decision-record table: each row is a fictional ADR at a different level of completeness.

Decision record Flags
Weak ("use multi-agent... obviously the best") 7 (six missing sections, absolute wording)
Partial 4 (no evidence, no revisit trigger, one alternative, no downside)
Complete 0

What to notice: "obviously the best" alone draws a flag for absolute wording, independent of the six missing sections. The linter treats overconfident language as its own red flag. The theory is that a decision stated as obvious is usually one nobody checked the numbers on.

{{svg:mix|Break-even on traffic mix, using illustrative scores.}} {{svg:matrix|Weighted scores, win probabilities under uncertain weights, and the margin.}}