Part 2 of 4 · 11 min read
Definitions
Covers: Definitions; The decision; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Fine-tuning
- Fine-tuning: continuing to train a pretrained model's weights on your own examples so it behaves differently.
Information-based vs behaviour-based failure
- Information-based: the model lacks, or has outdated, facts it needs. Behaviour-based: it has the facts but produces the wrong form, style, format or policy.
Supervised fine-tuning (SFT) and distillation
- SFT: training on (input, desired output) pairs. Distillation: training a small model to imitate a larger one's outputs.
Break-even
- Break-even volume: the request rate at which cumulative savings cover the up-front and recurring cost of tuning within a chosen horizon.
Base-model refresh
- Refresh: re-doing the tuning on a newer base model, a recurring cost.
The decision
Results
| Situation (decision.py) | Recommendation |
|---|---|
| No eval set | Build an evaluation set |
| 3 scored prompt experiments, behaviour failure | Iterate on prompts systematically |
| Outdated answers (information failure) | Add retrieval, start with term-based search |
| Rare DSL output, 1,500 examples, can host | Fine-tune a small model on curated examples |
| Format failures, 120 examples, cannot host | Remove blockers (data, serving), then fine-tune |
| Monthly requests (assumed economics) | Saving per month | Payback |
|---|---|---|
| 20,000 | -$80 | never |
| 50,000 | $1,000 | never (re-tunes eat it) |
| 100,000 | $2,800 | 8 months |
| 300,000 | $10,000 | 2 months |
| 1,000,000 | $35,200 | 1 month |
| Break-even for 12-month payback (assumed) | Requests per month |
|---|---|
| Base case | about 75,300 |
| With a 2-point quality loss | about 158,600 |
| With a re-tune every 3 months | about 105,600 |
| Months | Tuned model, never retrained | Retrieval |
|---|---|---|
| 0 | 95% | 86% |
| 6 | 74% | 86% |
| 12 | 56% | 86% |
| 18 | 43% | 86% |
| 24 | 33% | 86% |
| Labelled examples (toy task, 5 seeds) | In-context (16 exemplars) | Tuned |
|---|---|---|
| 16 | 60% | 69% |
| 64 | 59% | 75% |
| 256 | 60% | 88% |
| 1,024 | 60% | 92% |
| 2,048 | 61% | 92% |
- Most "we need to fine-tune" cases are prompt-experiment or retrieval cases until measured otherwise.
- Volume decides cost cases: below break-even, tuning is a loss even if it works.
- Facts in weights decay; form does not (unless the task definition changes).
- The toy curve's lesson is the shape: tuned accuracy keeps rising with data while a fixed context does not; it does not tell you how much data a real task needs.
Design rules
- Eval set first, then at least ten scored prompt variants before considering tuning.
- Classify every failure as information or behaviour and count each.
- Fix information failures with retrieval, starting simple (BM25) before embeddings.
- Tune for stable behaviour: strict format, rare syntax, a house style, a distilled step.
- Compute break-even with caching, upkeep and re-tune cost included.
- Keep a baseline: compare the tuned model against the best prompted model on the same eval, and on tasks it should not have changed.
- Check for regressions on neighbouring tasks (tuning can degrade them).
- Prefer a provider's tuning API over self-hosting when volume is modest, and accept its limits.
- Plan the refresh: record the data and recipe so re-tuning on a new base model is cheap.
- Keep facts out of the weights when the data changes often.
Interview questions, with the follow-ups
When would you fine-tune instead of using RAG or better prompts?
"When the failure is about form and I have shown that systematic prompting does not fix it: a strict output format, an uncommon syntax, a consistent style, or distilling a stable high-volume step into a small model. I need enough labelled examples, a way to serve the model, and volume above break-even. For missing or changing facts I use retrieval; if both problems exist I start with retrieval." Follow-up: "How do you know prompting is exhausted?" A tracked set of prompt variants scored on the eval, not three ad hoc attempts.
How would you decide whether tuning pays for itself?
"Per-request savings times volume, minus upkeep, against up-front data and training cost, with a re-tune each time a better base model arrives, and minus the value of any quality loss. In my model 75,000 requests a month paid back in a year; at 50,000 the re-tunes consumed the saving." Follow-up: "What changes it?" Caching discounts, cheaper base models, volume growth, and the quality gap.
Why not put the company's knowledge into the model by fine-tuning?
"Facts go stale in the weights, you cannot cite or delete them easily, and the study I recall found retrieval beat tuning on new facts. Tuning is for behaviour. In my simulation a tuned model fell from 95 to 56 percent in a year at 4 percent monthly change." Follow-up: "Can you do both?" Yes: tune for format and tone, retrieve for facts.
What can go wrong after fine-tuning?
"It can degrade other tasks, overfit to the data's quirks, be overtaken by a new base model, and add serving and maintenance load. I keep a regression eval on tasks the tuning should not touch and the baseline to compare against." Follow-up: "How do you detect overfitting?" Held-out examples from a different time or source, not a random split of the same data.
Common mistakes
- Tuning before building an eval set.
- Concluding prompting failed after a few tries.
- Using tuning to teach facts that change.
- Ignoring re-tune cost when a new base model arrives.
- Comparing with a weak baseline (a naive prompt, no retrieval).
- Ignoring caching in the per-request cost of the baseline.
- No regression check on other tasks.
- Random train/test split of near-duplicate examples.
- Assuming domain pretraining beats a general frontier model (see the BloombergGPT comparison above).
- Not counting data creation and upkeep as the main cost.
What the example does not do
- No model is tuned; the experiments are a decision procedure, an economic model and toys.
- The sample-efficiency toy uses a linear task and a nearest-neighbour stand-in for in-context learning; real in-context learning with a strong model can be far better than my stand-in.
- Staleness is a simple random-change model; real knowledge changes unevenly and tuned models retain some facts partially.
- The economics ignore GPU price variation, provider tuning prices, data privacy benefits and latency gains.
- Thresholds are rules of thumb from my judgement, not measured.
- Optional real experiment (I did not run this): take one failing step of your agent, score 12 prompt variants on a 100-example eval, classify the remaining failures as information or behaviour, and only then estimate break-even with your real volume and prices.
Check yourself
A teammate says "prompting didn't work, let's fine-tune". What do you ask first? (How many scored prompt variants, on what eval set, and are the remaining failures about missing facts or about form? The decision code refuses to recommend tuning with fewer than 10 scored experiments.)
For the curious Volume decides, and facts go stale in weights
With my assumed prices, 20,000 requests a month never paid back and 50,000 never did either (the $6,000 re-tune every 6 months ate the $1,000 monthly saving); 100,000 paid back in 8 months. A model tuned on facts that change 4 percent a month fell from 95 to 56 percent in a year, while retrieval stayed at 86. The sample-efficiency curve is a toy classifier analogy, not a language model.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Build the eval set first | Then score at least ten prompt variants. |
| Label each failure information or behaviour | Retrieval for the first, tuning for the second. |
| Do the break-even with your real numbers | Include caching, upkeep and re-tuning. |
| Keep a regression eval | On tasks the tuning should not change. |
Draft: confirm your provider's current tuning options and prices, and which tuned models you may host.
What you must be able to say (one page)
- The rule of thumb. A widely used rule of thumb: finetuning is for form, and RAG is for facts. If a failure is information-based (the model lacks or has outdated facts) add retrieval. If it is behaviour-based (wrong format, style, tone, rare syntax, refusal pattern) fine-tuning is a candidate. With both kinds of failure, start with retrieval, and start with simple term-based search (BM25).
- Order of attempts. Finetuning needs far more data, hardware and skill than prompting, so it is normally attempted after extensive prompt experiments, and the two can be combined. Prompt experiments are often minimal and unsystematic, so "prompting did not work" often means "I tried three prompts". My decision code (
decision.py) refuses to recommend tuning until there is an eval set and 10 or more scored prompt experiments (the 10 is my assumed threshold, not a standard). - Good reasons to tune: quality on a narrow task, strict structured output (JSON, YAML), uncommon syntax such as a rare SQL dialect, bias mitigation, and distilling a large model into a small one. Example: Grammarly's CoEdIT, a Flan-T5 model tuned on about 82,000 (instruction, output) pairs, beat a much larger GPT-3 variant (about 60 times bigger) on text editing. I am quoting this from memory and did not re-check the paper.
- Good reasons not to: tuning can degrade other tasks, needs data, talent, serving and maintenance, and a new base model may overtake your tuned one.
- Domain pretraining is not a guarantee. A published comparison put BloombergGPT behind GPT-4 zero-shot on two finance tasks: FiQA weighted F1 75.07 vs 87.15, ConvFinQA accuracy 43.41 vs 76.48. I am quoting those numbers from memory and did not re-run or re-check them.
- RAG beat tuning on fresh facts in a study I recall (Ovadia et al., 2023, "Fine-Tuning or Retrieval?"; not re-checked). My toy simulation shows the mechanism (all rates assumed): when 4 percent of facts change each month, a tuned model never retrained falls from 95 to 74 percent accuracy after 6 months, 56 after 12 and 33 after 24, while retrieval stays at 86 (0.90 recall x 0.95 reading, assumed). Retraining every 6 months holds 95 percent, but each retrain costs money.
- Tuning has a price, so do the break-even. In my model ($0.040 per request for a large prompted model, $0.004 for a tuned small one, $15,000 up front, $800 a month upkeep, $6,000 re-tune every 6 months when a better base model arrives; all assumed): 20,000 requests a month never pays back; 50,000 never pays back either (the monthly saving of $1,000 is eaten by the re-tune, $6,000 per 6 months); 100,000 pays back in 8 months; 300,000 in 2; 1,000,000 in 1. For a 12-month payback the break-even is about 75,000 requests a month; with a 2-point quality loss valued at $1,500 per point per month it is about 158,600; with a base-model refresh every 3 months it is about 105,600.
- Caching weakens the cost argument. Prompt caching makes the long prompt of the baseline cheaper, which shrinks the saving from tuning. Include your caching discount in the per-request prompted cost before you compute break-even.
- What to say in an interview: "I would not tune first. I would build an eval set, run systematic prompt experiments, classify the failures as information or behaviour, add retrieval for the first kind, and consider tuning for the second only if I have enough data, a way to serve the model and enough volume to pay back. Then I would compare the tuned model with the best prompted baseline on the same eval."
- Honest limits: the decision thresholds (10 experiments, 500 examples, 3 stable months) are my rules of thumb; every dollar figure and rate is assumed; the sample-efficiency curve is a toy classifier analogy, not a language model.