Interview questions and limits: Retrieval-based memory

Covers: Interview questions, with the follow-ups; Common mistakes; What the example does not do.

Interview questions, with the follow-ups

How do you retrieve memories for an agent?

"I keep memories outside the window with a key, a timestamp and an importance score assigned at write time. Before each call I embed the question and rank the memories by a blend of cosine similarity, recency decay and importance, then put the top k into the window under a token budget. The part I'd be careful about is currency: a vector store can't tell which of two similar memories is current, so I supersede on a key at write time and exclude superseded items from default search." Follow-up: "Why not just use the newest?" Recency alone returns recent chatter and misses old important facts: in my test it scored 0 percent, and a similarity-based search found the constraint written on day 2. Follow-up: "Why not raise k?" More neighbours do not remove a stale one; in my test similarity-only plateaued at 57 percent at k = 8 while supersession reached 100 percent.

Two memories contradict each other. What happens, and how do you prevent it?

"A vector store treats them as near-duplicates, so both are retrieved and the model sees a conflict it has no basis to resolve. I prevent it at write time: facts carry a key, and a new value supersedes the old one, which stays for audit but is excluded from search. For unkeyed notes I'd rely on timestamps and let the model see the dates, and I'd measure how often a stale value reaches the window." Follow-up: "What about facts that are both true?" Then there should not be a single key; a preference history ('used to prefer', 'now prefers') is a different record type, which is a schema decision, not a retrieval one.

How do you stop memory poisoning?

"Retrieval makes a poisoned write permanent and ranks it by relevance, so the control is at write time. Only the user or the system may write a keyed fact; everything else goes to quarantine and is logged. In my test a tool result that wrote 'meeting at 3am' replaced the real value 0 times out of 5 with the gate and 5 out of 5 without." Follow-up: "Is the gate enough?" No. It protects keyed facts; an unkeyed note from a web page can still be retrieved and read by the model, so retrieved text from untrusted sources still has to be fenced in the window (module 1).

How do you choose the weights and k?

"I don't guess. I build a labelled set of questions with the answer and the stale value for each, measure answer-retrieved and stale-retrieved separately, and sweep the weights and k. In my example the best setting put the recency weight near zero once supersession existed, which I would not have predicted. I'd also hold out part of the set so I'm not tuning to the examples I report."

Common mistakes

Treating relevance as correctness. The most similar memory is the one most like the question, not the one that is true now.
Keeping old and new values side by side and hoping the model picks the newer. It sees two plausible values with no way to know.
Relying on k to cover for bad ranking. It costs tokens and does not remove stale items.
Weighting recency heavily. It lets today's chatter bury a day-2 constraint.
Letting any source write a keyed fact. One poisoned write supersedes the real one for every future session.
Never deleting. A forgotten memory that is still indexed is still retrievable; delete from the index and its statistics.
Measuring only "answer present". Count the stale value separately; this experiment's whole result is in that gap.

What the example does not do

The embedding is an imitation; a real one has dense vectors, different failure modes (a paraphrase can score low; unrelated texts can score high) and a cost per call.
No approximate search. A real store uses an index (for example HNSW) and trades a little recall for speed; the numbers here are exact.
One answer per question, found by presence. Multi-hop questions, aggregation ("how many meetings this week") and temporal questions ("what was my preference in June?") are not tested.
Importance is assigned by me. A real system has to estimate it, with its own errors.
Duplicates (the same note written many times) are not collapsed; a real store needs deduplication or diversity-aware ranking so one note does not fill the top k.
Optional real-model experiment (I did not run this): replace embed.py with a real embedding model and re-run run_all.py. Expect paraphrase matching to behave differently and the exact percentages to change; the claim to test is whether similar-but-stale values still tie.

Check yourself

The store holds "I prefer meetings at 2pm" and, newer, "I prefer meetings at 11am". The top 3 results for "when do I like to schedule a call" include both. Does raising k to 8 help? (No. It adds more neighbours, and both values are still there. Supersede the old value at write time so only the current one is searchable.)

For the curious Why currency belongs at write time

Two similar facts have almost the same vector, so no ranking by similarity can say which one is current. The fix is a key on the fact and a rule that a newer write replaces an older one. In this example that took the answer rate from 60 to 90 percent at k = 3, and the embedding is an imitation, so treat the direction as the result, not the numbers.

What you will use daily at work

Practice	What it looks like
Key facts that change	`meeting_time`, `home_city`, `preferred_language`.
Blend relevance, recency, importance	Weights are your choice; test them on your own questions.
A trusted-source gate	Tool results cannot overwrite what the user said.

Draft: confirm which vector store and embedding model your team uses for memory.

What you must be able to say (one page)

Retrieval-based memory keeps memories outside the window and, before each model call, selects the few that are relevant to the current question and places them in the window. The two operations are write (store a record with metadata) and recall (rank and select).
A retrieval score usually blends three signals: relevance (vector similarity between the query and the memory), recency (a decay on age) and importance (a score assigned at write time). The weights are design choices, not constants of nature.
Similarity alone has a structural blind spot: it cannot tell which of two similar memories is current. "I prefer meetings at 2pm" and "I prefer meetings at 11am" are almost identical vectors. A vector store returns both, and the model sees a contradiction. The fix is not a better embedding; it is write-time supersession on a key, so only the current value is searchable.
Recency alone fails the other way: it returns the latest chatter and misses the old, important fact (0 percent in the example). Heavy recency weighting also hurts once supersession exists (53 percent at weight 1.0, 10 percent at 3.0, against 90 percent at 0.3).
Increasing k does not remove a stale value. It adds more neighbours. In the example, similarity-only plateaus at 57 percent at k = 8, while supersession reaches 100 percent.
Who may write is a retrieval concern too. A tool result that writes "meeting at 3am" under the same key supersedes the user's fact. A trusted-source gate (module 2) stops it: 100 percent safe with the gate, 0 percent without.
What you can quote: at k = 3 over 30 questions, recency-only 0 percent, similarity-only 43 percent, blended 60 percent, blended with supersession 90 percent; stale value retrieved on 30 percent of questions without supersession and 0 percent with it. (The embedding here is an imitation; see the box below.)