CSE Roadmap

Core tier · about 6 hours · 10 parts

Part 7 of 10 · 3 min read

Mistakes that let a stale memory win

Common mistakes, and what the example does not do.

Mistakes that let a stale memory win

  • Treating relevance as correctness. The most similar memory is the one most like the question, not the one that is true now.
  • Keeping old and new values side by side and hoping the model picks the newer. It sees two plausible values with no way to know.
  • Relying on k to cover for bad ranking. It costs tokens and does not remove stale items.
  • Weighting recency heavily. It lets today's chatter bury a day-2 constraint.
  • Letting any source write a keyed fact. One poisoned write supersedes the real one for every future session.
  • Never deleting. A forgotten memory that is still indexed is still retrievable; delete from the index and its statistics.
  • Measuring only "answer present". Count the stale value separately; this experiment's whole result is in that gap.

What the example does not do

About this example What the example does not do
  • The embedding is an imitation; a real one has dense vectors, different failure modes (a paraphrase can score low; unrelated texts can score high) and a cost per call.
  • No approximate search. A real store uses an index (for example HNSW) and trades a little recall for speed; the numbers here are exact.
  • One answer per question, found by presence. Multi-hop questions, aggregation ("how many meetings this week") and temporal questions ("what was my preference in June?") are not tested.
  • Importance is assigned by hand. A real system has to estimate it, with its own errors.
  • Duplicates (the same note written many times) are not collapsed; a real store needs deduplication or diversity-aware ranking so one note does not fill the top k.
  • Optional real-model experiment: replace embed.py with a real embedding model and re-run run_all.py. Expect paraphrase matching to behave differently and the exact percentages to change; the claim to test is whether similar-but-stale values still tie.

Check yourself

The store holds "I prefer meetings at 2pm" and, newer, "I prefer meetings at 11am". The top 3 results for "when do I like to schedule a call" include both. Does raising k to 8 help?

Show answer

No. It adds more neighbours, and both values are still there. Supersede the old value at write time so only the current one is searchable.

For the curious Why currency belongs at write time

Two similar facts have almost the same vector, so no ranking by similarity can say which one is current. The fix is a key on the fact and a rule that a newer write replaces an older one. In this example that took the answer rate from 60 to 90 percent at k = 3, and the embedding is an imitation, so treat the direction as the result, not the numbers.

What you will use daily at work

Practice What it looks like
Key facts that change meeting_time, home_city, preferred_language.
Blend relevance, recency, importance Weights are your choice; test them on your own questions.
A trusted-source gate Tool results cannot overwrite what the user said.