Part 6 of 13 · 4 min read
What the example found
The traffic-mix break-even, the design-variant comparison, weight-uncertainty win rates, the critical weights, the scaling pitfall, and the decision-record linter.
How to read the break-even table: each row is a different assumed share of tool-heavy traffic; the value column is how much a quality point would have to be worth for the gain to justify the token cost.
| Share of tool-heavy traffic | Direct score | Multi-agent score | Gain | Value per point needed (assumed $1 per million tokens) |
|---|---|---|---|---|
| 5% | 9.55 | 9.30 | -0.26 | never |
| 10% | 9.41 | 9.29 | -0.12 | never |
| 20% | 9.12 | 9.28 | 0.16 | $0.094 |
| 30% | 8.83 | 9.27 | 0.44 | $0.034 |
| 50% | 8.25 | 9.25 | 1.00 | $0.015 |
| 70% | 7.67 | 9.23 | 1.56 | $0.010 |
| 90% | 7.09 | 9.21 | 2.12 | $0.007 |
What to notice: the required value per point drops fast as traffic mix rises. At 90% tool-heavy traffic, even a quality point worth less than a cent justifies the switch. Below the ~14% break-even, the multi-agent system never clears the bar, no matter how cheap a quality point is.
How to read the design-variant table: each row is one of five architecture candidates from module 3's simulation, scored against a fixed weighted scale.
| Design variant (module 3 sim., assumed quality) | p95 (s) | Cost | Quality | Weighted score | Passes p95 <= 8 s and q >= 0.72 |
|---|---|---|---|---|---|
| As designed | 10.4 | $0.0125 | 0.78 | 0.347 | no (p95) |
| Cap loop at 2 | 9.1 | $0.0115 | 0.77 | 0.414 | no (p95) |
| Small planner | 7.7 | $0.0063 | 0.75 | 0.515 | yes |
| Small planner + cap | 7.0 | $0.0061 | 0.73 | 0.520 | yes |
| No router (big model only) | 11.3 | $0.0156 | 0.84 | 0.377 | no (p95) |
What to notice: the two variants with the highest raw quality ("as designed" and "no router") both fail the latency requirement. They are eliminated before the weighted score even matters. The eventual top weighted score (0.520) belongs to the variant with the lowest quality among the five. That only looks surprising until you remember the constraint filter already removed its higher-quality competitors.
How to read the weight-confidence table: each row is a level of confidence (concentration) in the stated base weights; the columns are each surviving variant's probability of winning across simulated weight draws.
| Weight confidence (concentration) | Small planner wins | Small planner + cap wins |
|---|---|---|
| 5 | 45% | 55% |
| 20 | 40% | 60% |
| 50 | 35% | 65% |
| 200 | 23% | 77% |
What to notice: even at the highest confidence level tested, the losing option still wins 23% of the time across simulated weight draws. Uncertainty in the weights never fully resolves for a comparison this close. That is the table's whole point: it reports a probability, not a guarantee.
How to read the sensitivity table: each row is a different check on how fragile the winning decision is.
| Sensitivity of the choice | Result |
|---|---|
| With 10% noise on the raw values (confidence 20) | 45% / 55% |
| p95 weight below | 0.37 flips to the cap variant |
| Quality weight above | 0.43 flips to the plain small planner |
| Cost weight | never flips it |
| Winner's quality: assumed vs break-even | 0.73 vs 0.728 |
| All five options, weights uncertain (confidence 20) | winner 60%, small planner 30%, big model only 10%, the other two 0% |
What to notice: the winner's assumed quality (0.73) sits just 0.002 above the point where it would lose (0.728). A margin this thin is a direct signal to go measure the real quality number rather than keep debating the weights. The module states that as its own explicit rule.
How to read the normalisation-pitfall table: each row is a different scaling choice or option set at fixed weights 0.6/0.4.
| Normalisation pitfall (weights 0.6 / 0.4) | Winner |
|---|---|
| Min-max, options A and B | A |
| Min-max, adding dominated option D | B |
| Fixed targets, with or without D | B |
What to notice: adding option D flips the min-max winner from A to B simply by stretching the scale. Option D should never be chosen anyway, since it is dominated. The fixed-target scaling gives the same answer (B) whether or not D is present at all. This is the rank reversal made concrete.
How to read the decision-record table: each row is a fictional ADR at a different level of completeness.
| Decision record | Flags |
|---|---|
| Weak ("use multi-agent... obviously the best") | 7 (six missing sections, absolute wording) |
| Partial | 4 (no evidence, no revisit trigger, one alternative, no downside) |
| Complete | 0 |
What to notice: "obviously the best" alone draws a flag for absolute wording, independent of the six missing sections. The linter treats overconfident language as its own red flag. The theory is that a decision stated as obvious is usually one nobody checked the numbers on.
{{svg:mix|Break-even on traffic mix, using illustrative scores.}} {{svg:matrix|Weighted scores, win probabilities under uncertain weights, and the margin.}}