Part 6 of 10 · 2 min read
Answering retrieval-memory questions in an interview
How do you retrieve memories for an agent?
"I keep memories outside the window with a key, a timestamp and an importance score assigned at write time. Before each call I embed the question and rank the memories by a blend of cosine similarity, recency decay and importance, then put the top k into the window under a token budget. The part I'd be careful about is currency: a vector store can't tell which of two similar memories is current, so I supersede on a key at write time and exclude superseded items from default search." Follow-up: "Why not just use the newest?" Recency alone returns recent chatter and misses old important facts: in the example's test it scored 0 percent, and a similarity-based search found the constraint written on day 2. Follow-up: "Why not raise k?" More neighbours do not remove a stale one; in the example's test similarity-only plateaued at 57 percent at k = 8 while supersession reached 100 percent.
Two memories contradict each other. What happens, and how do you prevent it?
"A vector store treats them as near-duplicates, so both are retrieved and the model sees a conflict it has no basis to resolve. I prevent it at write time: facts carry a key, and a new value supersedes the old one, which stays for audit but is excluded from search. For unkeyed notes I'd rely on timestamps and let the model see the dates, and I'd measure how often a stale value reaches the window." Follow-up: "What about facts that are both true?" Then there should not be a single key; a preference history ('used to prefer', 'now prefers') is a different record type, which is a schema decision, not a retrieval one.
How do you stop memory poisoning?
"Retrieval makes a poisoned write permanent and ranks it by relevance, so the control is at write time. Only the user or the system may write a keyed fact; everything else goes to quarantine and is logged. In the example's test a tool result that wrote 'meeting at 3am' replaced the real value 0 times out of 5 with the gate and 5 out of 5 without." Follow-up: "Is the gate enough?" No. It protects keyed facts; an unkeyed note from a web page can still be retrieved and read by the model, so retrieved text from untrusted sources still has to be fenced in the window (module 1).
How do you choose the weights and k?
"I don't guess. I build a labelled set of questions with the answer and the stale value for each, measure answer-retrieved and stale-retrieved separately, and sweep the weights and k. In the example's example the best setting put the recency weight near zero once supersession existed, which I would not have predicted. I'd also hold out part of the set so I'm not tuning to the examples I report."