Part 9 of 12 · 2 min read
Mistakes that waste the data-and-serving investment
Common mistakes, and what the example does not do.
Mistakes that waste the data-and-serving investment
- Tuning before building an eval set.
- Concluding prompting failed after a few tries.
- Using tuning to teach facts that change.
- Ignoring re-tune cost when a new base model arrives.
- Comparing with a weak baseline (a naive prompt, no retrieval).
- Ignoring caching in the per-request cost of the baseline.
- No regression check on other tasks.
- Random train/test split of near-duplicate examples.
- Assuming domain pretraining beats a general frontier model (see the domain-pretraining note above).
- Not counting data creation and upkeep as the main cost.
What the example does not do
About this example What the example does not do
- No model is tuned; the experiments are a decision procedure, an economic model and toys.
- The sample-efficiency toy uses a linear task and a nearest-neighbour stand-in for in-context learning; real in-context learning with a strong model can be far better than the stand-in.
- Staleness is a simple random-change model; real knowledge changes unevenly and tuned models retain some facts partially.
- The economics ignore GPU price variation, provider tuning prices, data privacy benefits and latency gains.
- Thresholds are rules of thumb from the judgement, not measured.
- Optional real experiment: take one failing step of your agent, score 12 prompt variants on a 100-example eval, classify the remaining failures as information or behaviour, and only then estimate break-even with your real volume and prices.
Check yourself
A teammate says "prompting didn't work, let's fine-tune". What do you ask first?
Show answer
How many scored prompt variants, on what eval set, and are the remaining failures about missing facts or about form? The decision code refuses to recommend tuning with fewer than 10 scored experiments.
For the curious Volume decides, and facts go stale in weights
With the assumed prices, 20,000 requests a month never paid back and 50,000 never did either (the $6,000 re-tune every 6 months ate the $1,000 monthly saving); 100,000 paid back in 8 months. A model tuned on facts that change 4 percent a month fell from 95 to 56 percent in a year, while retrieval stayed at 86. The sample-efficiency curve is a toy classifier analogy, not a language model.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Build the eval set first | Then score at least ten prompt variants. |
| Label each failure information or behaviour | Retrieval for the first, tuning for the second. |
| Do the break-even with your real numbers | Include caching, upkeep and re-tuning. |
| Keep a regression eval | On tasks the tuning should not change. |