Where training data comes from, and what can go wrong.
Data
How much? In the example, question side only:
Training pairs
New questions
General language
10
4%
32%
20
7%
35%
40
20%
40%
72
48%
50%
With too few pairs the model memorises the pairs it saw, forgets general language, and gains nothing on new questions. Your numbers will differ; the lesson is that the gain needs coverage: every term the model should learn must appear in several training pairs.
Where pairs come from:
Logs and feedback: questions with the page a user clicked, or the document a support agent attached. The most realistic data, and biased toward what your current system already shows.
Existing pairs: ticket and resolution, title and body, question and accepted answer.
Generated questions: a language model writes plausible questions for each passage. Cheap and wide, but they sound like a model, not like your users; check a sample by hand.
Labelling with a stronger model (cross-encoder or language model) to expand a small labelled set.
Split by document or topic before training, not by random row, or near-copies of training pairs leak into the test (module 7). Keep a held-out set and a general-language canary set that you never train on.
Risks and defences
Risk
What it looks like
Defence
Forgetting
general and out-of-domain questions get worse
gentler learning rate, mix in general pairs, keep the tuned part small (adapter), a canary set
Memorising
near-perfect training score, much lower on new questions
held-out validation by topic, more varied data, fewer epochs
Narrow transfer
no gain on topics absent from training
cover the vocabulary; plan retraining when the domain grows
False negatives
mined "hard negatives" teach wrong distinctions
score thresholds or rank windows, human check on a sample
Evaluation leakage
great test score, disappointing real use
split by topic, test on logged real questions
Maintenance
the domain changes, the model does not
version the model, schedule re-evaluation, re-embed through blue-green (module 9)
What worked best in the example (question side only; new questions and general language):
Setting
New, same problems
General language
default (learning rate 0.02)
50%
51%
gentler (learning rate 0.005)
55%
72%
pull the matrix back toward the original (weight 0.01)
40%
66%
replay 20 general-language pairs
47%
77%
The cheap fix was a gentler learning rate, which was better on both columns than the default, and mixing in general pairs protected general ability most. Pulling toward the original cost domain accuracy. Differences of a few points between rows are within run-to-run noise.
New-question accuracy (dark bars) against general-language accuracy (light bars) for five settings