Part 7 of 12 · 1 min read
How to build it so it holds up
- Eval set first, then at least ten scored prompt variants before considering tuning.
- Classify every failure as information or behaviour and count each.
- Fix information failures with retrieval, starting simple (BM25) before embeddings.
- Tune for stable behaviour: strict format, rare syntax, a house style, a distilled step.
- Compute break-even with caching, upkeep and re-tune cost included.
- Keep a baseline: compare the tuned model against the best prompted model on the same eval, and on tasks it should not have changed.
- Check for regressions on neighbouring tasks (tuning can degrade them).
- Prefer a provider's tuning API over self-hosting when volume is modest, and accept its limits.
- Plan the refresh: record the data and recipe so re-tuning on a new base model is cheap.
- Keep facts out of the weights when the data changes often.