Part 5 of 6 · 3 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-recall, run pytest -q (about a second), python demo.py and python run_all.py.
Done when: you report, from results.json, the correct and stale-present rates for similarity and blended+supersede, and explain in one sentence why the two facts in the demo have the same similarity.
Hint
E.table() returns the per-policy dictionary used in the notes.
Easy 2: Add a synonym (30 minutes)
Add "cuisine" to the food words in embed.py. Write a test that a question using "cuisine" now matches "My diet is vegan.", and one that an unrelated question still scores 0.
Done when: both tests pass and you say what a real embedding model would do for an unseen word instead.
Hint
SYN maps a word to its concept; the question must not share any literal word with the stored text.
Easy 3: Move k (30 minutes)
Run E.run("similarity", k=k) and E.run("blended+supersede", k=k) for k from 1 to 10.
Done when: you plot or tabulate both and find the smallest k at which supersession reaches 95 percent, and say why similarity never gets there.
Hint
E.run(policy, k=k)["correct"].
Medium
Medium 1: Deduplicate on write (2 hours)
Make Store.add skip a note whose cosine similarity to an existing note is above 0.95. Add a world option that repeats each noise note three times.
Done when: you show the number of items stored with and without deduplication, that top-3 contains three different notes, and that no keyed fact is lost.
Hint
compute embed.cosine against every existing item in add; compare to 0.95.
Medium 2: Learn the weights (3 hours)
Write a grid search over Weights on worlds with seeds 100 to 109 (tuning) and report on seeds 0 to 4 (test).
Done when: you report the best weights, the gap between tuning and test scores, and say whether recency came out near zero.
Hint
E.run(policy, seeds=range(100, 110), w=Weights(...)).
Medium 3: Facts that are both true (3 hours)
Add a second record type, a preference history, for facts where the past value still matters ("I used to prefer 2pm"). Retrieval for "what did I prefer in June?" must return the old value; "now?" must not.
Done when: both questions are answered correctly, supersession still hides the old value for "now", and you say how the query tells the two cases apart.
Hint
keep superseded items in the store and filter by day when the query contains a date.
Hard
Hard 1: A real embedding model (4 hours)
Replace embed.py with a real embedding model and re-run the experiment. (I did not run this, so your numbers are the ground truth.)
Done when: you report the similarity of the two stale/current facts, the correct rate for each policy, the model name and the date, and say whether anything in my conclusion changed.
Hint
keep vectorise returning a dict, or change cosine to dense vectors and update the tests.
Hard 2: Extractor errors (4 hours)
Replace the keys supplied by the world with a model-based key extractor and measure how often it assigns the same key to the old and new value. (I did not run this.)
Done when: you report the rate of missed supersession and of wrong supersession (a different fact overwritten), and which one is worse.
Hint
the gate and the supersession code stay the same; only key comes from a different place.
Hard 3: Temporal and multi-fact questions (5 hours)
Add questions that need two memories ("which day is the report due and who gets it?") and one that needs the value as of a date. (I did not run this.)
Done when: you report which policies fail each type, say what retrieval would need (multi-query, filters by date), and add a test for each.
Hint
for "as of", add valid_from and valid_to to Item rather than relying on superseded_by.