CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 8 of 12 · 2 min read

Answering fine-tuning questions in an interview

When would you fine-tune instead of using RAG or better prompts?

"When the failure is about form and I have shown that systematic prompting does not fix it: a strict output format, an uncommon syntax, a consistent style, or distilling a stable high-volume step into a small model. I need enough labelled examples, a way to serve the model, and volume above break-even. For missing or changing facts I use retrieval; if both problems exist I start with retrieval." Follow-up: "How do you know prompting is exhausted?" A tracked set of prompt variants scored on the eval, not three ad hoc attempts.

How would you decide whether tuning pays for itself?

"Per-request savings times volume, minus upkeep, against up-front data and training cost, with a re-tune each time a better base model arrives, and minus the value of any quality loss. In the example's model 75,000 requests a month paid back in a year; at 50,000 the re-tunes consumed the saving." Follow-up: "What changes it?" Caching discounts, cheaper base models, volume growth, and the quality gap.

Why not put the company's knowledge into the model by fine-tuning?

"Facts go stale in the weights, you cannot cite or delete them easily, and the study I recall found retrieval beat tuning on new facts. Tuning is for behaviour. In the example's simulation a tuned model fell from 95 to 56 percent in a year at 4 percent monthly change." Follow-up: "Can you do both?" Yes: tune for format and tone, retrieve for facts.

What can go wrong after fine-tuning?

"It can degrade other tasks, overfit to the data's quirks, be overtaken by a new base model, and add serving and maintenance load. I keep a regression eval on tasks the tuning should not touch and the baseline to compare against." Follow-up: "How do you detect overfitting?" Held-out examples from a different time or source, not a random split of the same data.