Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-finetune.
Done when: you can explain each recommendation in the decisions table.
Easy 2: Classify failures (30 minutes)
Take 20 failing outputs from an agent you built and label each information or behaviour.
Done when: you have counts and a next step for each kind.
Hint
if the answer is correct but wrongly formatted, it is a behaviour failure.
Easy 3: Your numbers (30 minutes)
Replace the Plan defaults with your prices and volume. (base_refresh_months must be a whole number.)
Done when: you report payback months and the break-even volume.
Medium
Medium 1: Prompt experiment log (3 hours)
Score 12 prompt variants on an eval set and keep a log.
Done when: you can say whether prompting is exhausted and why.
Hint
change one thing per variant so you can attribute gains.
Medium 2: Add caching (2 hours)
Add a cache discount to the prompted cost in economics.py and recompute break-even.
Done when: you report how much the break-even rises.
Hint
caching usually lowers the prompted cost per request, which raises the break-even volume.
Medium 3: Staleness with partial memory (3 hours)
Let a retrained model keep a share of old facts and let changes arrive unevenly.
Done when: you show how the gap to retrieval changes.
Hard
Hard 1: Tune a small model (8 hours)
Tune a small open model on one agent step and compare with the best prompted baseline and a regression eval. (I did not run this, so your numbers are the ground truth.)
Done when: you report quality, cost per request and break-even with real numbers.
Hint
hold out examples from a later period than the training data.
Hard 2: RAG versus tuning on fresh facts (8 hours)
Reproduce the idea of that study on a small corpus that changes. (I did not run this.)
Done when: you report accuracy over several update rounds for both methods.
Hard 3: Real sample efficiency (6 hours)
Tune on 100, 1,000 and 10,000 examples and plot quality. (I did not run this.)
Done when: you can say where returns diminish for your task.