Part 2 of 12 · 2 min read
Classify the failure before reaching for fine-tuning
"Prompting didn't work" usually means "I tried three prompts," and most failures turn out to need retrieval, not a tuned model.
Think of a new hire who keeps getting an answer wrong. Before sending them to a training course, check whether they actually have the information they need. Or they may have it but keep presenting it the wrong way.
In plain words: a decision procedure should refuse to recommend fine-tuning until there's an eval set and real prompt experiments behind it. Even then, it should route information failures to retrieval, and only behaviour failures toward tuning.
A worked example. With no eval set, the recommendation is simply to build one. Every other choice is a guess without it. With an eval set but only 3 scored prompt experiments and a behaviour failure, the recommendation is to iterate on prompts systematically, not to tune. Once prompting is genuinely exhausted, outdated answers (an information failure) get routed to retrieval, starting with term-based search. A rare DSL output with 1,500 labelled examples and hosting available gets routed to fine-tuning a small model. The same kind of failure with only 120 examples and no hosting gets routed to "remove the blockers, then fine-tune" instead.
The key lines. From code/mini-finetune/decision.py:
# from code/mini-finetune/decision.py
if not s.has_eval_set:
return dict(next_step="build an evaluation set", reasons=["every later choice is a guess without one"], blockers=["no eval set"])
if s.prompt_iterations < MIN_PROMPT_ITERATIONS:
return dict(next_step="iterate on prompts systematically", reasons=[f"only {s.prompt_iterations} scored prompt experiments; unsystematic prompting is the usual reason tuning looks necessary"], blockers=["prompting under-explored"])
Read it as: the function returns early, before even looking at the failure type, whenever the eval set or prompt-experiment count is too thin. It structurally cannot recommend tuning to a team that hasn't done the groundwork first.
Fine-tuning, information-based vs. behaviour-based failure
Fine-tuning is continuing to train a pretrained model's weights on your own examples so it behaves differently. An information-based failure means the model lacks or has outdated facts; a behaviour-based failure means it has the facts but produces the wrong form, style, format or policy.
Example: the DSL-output case above is behaviour-based. The model knows what a DSL is, it just doesn't emit the right syntax. That is exactly the kind of failure this decision procedure routes toward tuning rather than retrieval.