CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Eval set first, then at least ten scored prompt variants before considering tuning.
  • Classify every failure as information or behaviour and count each.
  • Fix information failures with retrieval, starting simple (BM25) before embeddings.
  • Tune for stable behaviour: strict format, rare syntax, a house style, a distilled step.
  • Compute break-even with caching, upkeep and re-tune cost included.
  • Keep a baseline: compare the tuned model against the best prompted model on the same eval, and on tasks it should not have changed.
  • Check for regressions on neighbouring tasks (tuning can degrade them).
  • Prefer a provider's tuning API over self-hosting when volume is modest, and accept its limits.
  • Plan the refresh: record the data and recipe so re-tuning on a new base model is cheap.
  • Keep facts out of the weights when the data changes often.