CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 6 of 12 · 3 min read

What the example found

The decision table, the break-even numbers, the staleness curve and the sample-efficiency curve.

How to read the decision table: each row is an independent situation fed to recommend(); the recommendation follows directly from which gate the situation fails first.

Situation (decision.py) Recommendation
No eval set Build an evaluation set
3 scored prompt experiments, behaviour failure Iterate on prompts systematically
Outdated answers (information failure) Add retrieval, start with term-based search
Rare DSL output, 1,500 examples, can host Fine-tune a small model on curated examples
Format failures, 120 examples, cannot host Remove blockers (data, serving), then fine-tune

What to notice: the last two rows are the same kind of failure (behaviour-based) with different amounts of data and hosting capacity. The recommendation changes not because the diagnosis changes, but because of what's actually available to act on it.

How to read the payback table: each row is the same cost model at a different monthly request volume; read "never" as "the recurring costs outpace the saving at this volume, not just slowly."

Monthly requests (assumed economics) Saving per month Payback
20,000 -$80 never
50,000 $1,000 never (re-tunes eat it)
100,000 $2,800 8 months
300,000 $10,000 2 months
1,000,000 $35,200 1 month

What to notice: 20,000 requests actually loses money per month (a negative saving), while 50,000 shows a positive monthly saving but still never pays back. These are two different failure modes, not the same one at different severities.

How to read the break-even table: all three rows target the same 12-month payback horizon; only one assumption changes per row.

Break-even for 12-month payback (assumed) Requests per month
Base case about 75,300
With a 2-point quality loss about 158,600
With a re-tune every 3 months about 105,600

What to notice: a 2-point quality loss roughly doubles the required volume. A tuned model that's slightly worse than the large prompted model isn't just a quality trade, it's a much higher bar to justify financially.

How to read the staleness table: both columns face the same 24-month period and the same 4 percent monthly change rate; only the method (retrain never vs. retrieval) differs.

Months Tuned model, never retrained Retrieval
0 95% 86%
6 74% 86%
12 56% 86%
18 43% 86%
24 33% 86%

What to notice: the tuned model starts 9 points ahead of retrieval and ends up 53 points behind it. The crossover happens well before month 12, which is exactly why "tune once and leave it" is a bad plan for facts that actually change.

How to read the sample-efficiency table: both columns face the same toy task and the same labelled-example counts; the in-context column is capped at 16 exemplars regardless of the row.

Labelled examples (toy task, 5 seeds) In-context (16 exemplars) Tuned
16 60% 69%
64 59% 75%
256 60% 88%
1,024 60% 92%
2,048 61% 92%

What to notice: the in-context column barely moves across the entire table (59-61 percent) no matter how much labelled data exists, because it's structurally capped at 16 exemplars. This is the shape of the trade-off, not a claim about how any particular real model's in-context learning behaves.

{{svg:decision|Prompt first, retrieve for facts, tune for behaviour.}} {{svg:economics|Payback by volume and knowledge going stale in the weights.}}