Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-tune, run pytest and python demo.py. From section 2 of the demo, give the share of new same-problem questions answered right before and after tuning (question side only), and the general-language share before and after.
Done when: you report four numbers and one sentence on what the drop in general language means for a real system.
Hint
use the test_seen and general columns of the "question side only" row.
Easy 2: Spot the gap (15 minutes)
For each pair say whether a general model is likely to handle it without tuning, and why: (a) "can't sign in" and "authentication failure"; (b) "refund" and "refund"; (c) "bill" and "invoice"; (d) "SKU-4471" and "the red mug".
Done when: you have four verdicts, and you say which ones keyword search (module 3) or a synonym list might fix more cheaply than training.
Hint
if the same words appear on both sides, an ordinary model already matches them.
Easy 3: Easy or hard negative (15 minutes)
For the question "How many seats does the Starter plan include?" label each candidate page as easy, semi-hard, hard or false negative: (a) a page about office parking; (b) "The Pro plan includes 10 seats"; (c) "Starter plans come with 1 seat" (the real answer, but not the page marked correct); (d) a page on Starter pricing.
Done when: you have four labels and you explain why (c) is dangerous in training.
Hint
a false negative is a page that really answers the question but was not labelled as the answer.
Medium
Medium 1: Your own domain (60 minutes)
Replace the lexicon with 12 jargon pairs from a field you know and regenerate the data. Report the untuned and tuned accuracy on new questions, and the effect of removing half the training pairs.
Done when: you have a table of three settings and one paragraph on which of your pairs it learned best and worst.
Hint
keep the same number of shared and different-word problems, or the gap changes.
Medium 2: Find the safe learning rate (45 minutes)
Sweep the learning rate (0.001, 0.003, 0.01, 0.03, 0.1) for the question-side model and plot new-question accuracy against general-language accuracy. Pick the setting you would ship and justify it, and say what you would monitor after shipping.
Done when: you have five points, a chosen setting with a reason, and a rule such as "reject the model if general accuracy falls below X".
Hint
pick the learning rate where general accuracy has not yet collapsed.
Medium 3: False negatives (60 minutes)
Duplicate some documents (same content, different filler words) so that some mined "hard negatives" are really correct. Measure how training with mined negatives behaves as the share of false negatives grows to 0%, 10% and 30%, then add a filter (skip candidates whose score is within a margin of the positive) and measure again.
Done when: you have a table over three levels with and without the filter, and a note on how you would estimate the false-negative rate in a real corpus.
Hint
a margin filter drops candidates that score nearly as high as the labelled answer.
Hard
Hard 1: A real embedding model (150 minutes)
With a sentence-embedding library and a small pretrained model, fine-tune on at least 300 pairs from a domain of your own (a documentation set, a FAQ, a legal or medical glossary you may use). Compare retrieval hit@5 on 60 held-out questions before and after, add a 40-question general canary set, and report any drop. (I did not run this, so your numbers are the ground truth.)
Done when: you have a before-and-after table for domain and canary questions, a note on training time and cost, and a decision: ship, adapt, or do not ship.
Hint
split the pairs by document, not by row, before training.
Hard 2: Query-side adapter in a live pipeline (120 minutes)
Put a small trained adapter on the query vector only of a real embedding model, leaving the stored document vectors untouched. Train it from logged or generated pairs, then compare it with full fine-tuning on accuracy, forgetting, training time and rollout effort.
Done when: you have both approaches measured on the same held-out set, with a table of cost (hours, re-embedded chunks) and a recommendation.
Hint
a linear layer applied to the query vector is enough for a first version.
Hard 3: Cover the unseen (120 minutes)
Section 5 showed no transfer to topics never in training. Design and test a way to improve unseen-topic accuracy: for example generating synthetic pairs for new terms with a language model, adding a glossary-derived synonym expansion at query time, or retraining on a schedule. Measure it on topics held out from training.
Done when: you show unseen-topic accuracy before and after your method, what it cost, and where it still fails.
Hint
held-out topics must stay out of both the training data and your synonym list.