Part 4 of 6 · 2 min read
Practical checklist
A short list to run before you trust a custom model.
- Exhaust cheaper fixes and measure the gap on real questions first (module 7).
- Build the data: pairs with coverage of your vocabulary, a topic-level split, a general canary set.
- Start from a pretrained embedding model and a small learning rate; consider a question-side adapter first.
- Compare against the untuned model on the same held-out questions, including general ones.
- Mine hard negatives carefully and check a sample for false negatives.
- Plan the rollout: second index, golden questions, switch both sides together, keep the old one (module 9).
- Re-evaluate on a schedule; version the model with its data.
Interview angle
"Our retrieval misses questions because users and documents use different words. Do we need a custom embedding model?" Probably not yet: first check chunking, add keyword search, try query rewriting or a reranker, and measure what remains on real questions. If a vocabulary gap persists and you have thousands of (question, passage) pairs, fine-tune a pretrained embedding model with a contrastive loss and in-batch plus carefully mined hard negatives, and evaluate against the untuned model on held-out questions split by topic, plus a general-language canary set because tuning can degrade general ability. Mention operations: if you tune both sides every vector must be re-embedded into a second index and switched together; a question-side adapter avoids that. Close with the limits: it will not transfer to topics absent from the training data, so you must maintain it as the domain changes.
Check yourself
Training accuracy reaches 100 percent while accuracy on general questions falls. What do you check before shipping?
Show answer
The held-out domain questions the model never saw, and the general set. A rise on training data proves nothing. A fall on general questions is the cost to weigh.
For the curious How ranks are counted
The evaluation ranks a document by counting how many documents score strictly higher than the right one, so a tie counts in the model's favour. With real, continuous scores ties are rare, but if two documents have identical text they share a score, so remove duplicates from a test corpus before you evaluate.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Trying cheaper fixes first | Better chunks, hybrid search, query rewriting and a reranker come before any training. |
| A held-out test set | Questions the model never trained on, plus a general set to catch forgetting. |
| Library support | Sentence-embedding libraries provide the loss and training loop; the work is the data. |