Part 6 of 7 · 4 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-compact, run pytest -q (about 50 seconds), then python demo.py and python run_all.py.
Done when: you report, from results.json, the accuracy and the summariser tokens for rolling and structured, say in one sentence why the rolling window is smaller than the structured one, and quote the first line of the rolling summary printed by demo.py.
Hint
demo.py prints the state block for each strategy; the first line of the rolling summary is the first line under "Summary of earlier work:".
Easy 2: Find the drift (40 minutes)
Run evaluate.simulate("rolling", seed) for seeds 0 to 4 and print drift_hits. Then run summarise.Summariser(summarise.Rates(1.0, 0.0, 1.0)) on ["Noted: the staging bucket is stg-eu-1234."].
Done when: you show the corrupted identifier, state which probe category a drifted identifier fails, and explain why the judge calls it missing rather than wrong.
Hint
Summariser.drifted maps original to corrupted tokens; the judge only counts stale and forbidden strings as wrong.
Easy 3: Change the budget (40 minutes)
Run E.aggregate("rolling", budget=b) and E.aggregate("truncate", budget=b) for b in 1,000, 2,000, 4,000 and 8,000.
Done when: you tabulate accuracy and rounds, find the smallest budget at which rolling passes 80 percent (or say none does), and explain in terms of rounds why it moves.
Hint
pass budget= through simulate; the chunk and cap fractions scale with it, so rounds fall as the budget rises.
Medium
Medium 1: Close the three holes (2 hours)
The mutation table lists the defects that escaped. Write one test for each, fix the code if needed, and re-run python mutants.py.
Done when: every mutant is caught, you show which new test catches each, and you say which hole you would have found without the mutation run.
Hint
the default-gate test calls Sim("structured") with no gate and feeds a tool turn; the "newest token" test needs a budget where the size is exactly equal; for the chunk-size mutant, assert on the number of rounds.
Medium 2: Preserve the original question (3 hours)
Add a preserved list to Sim that is never evicted or summarised, and put a task statement in it at step 1. Re-run the strategy table.
Done when: you report how much each of truncate, clear_tool and rolling changes, say whether the ranking of strategies changes, and explain why the task statement was cheap to preserve but the mid-conversation decisions are not.
Hint
add the preserved turns to state_block() or to the start of window() and subtract their tokens from the budget before FIFO.
Medium 3: Reversible stubs (3 hours)
Make clear_tool keep the cleared text in a side store keyed by step, and add a restore(step) method that the agent can call when a probe needs it. Add a probe that needs a cleared value (a number from a tool log).
Done when: the probe fails without restore and passes with it, you count the extra tokens that the restore adds to that step, and you say what makes a tool unsafe to re-run.
Hint
store {step: original_text} in _clear_tools; restore should append the text as a new tool turn and count the tokens.
Hard
Hard 1: A real summariser (4 hours)
Replace summarise.Summariser with a call to a real model that summarises a list of lines under a token cap. Run the rolling strategy on the five tasks. (I did not run this, so your numbers are the ground truth.)
Done when: you report the per-line keep rate for the salient lines you measured, the number of drifted identifiers, the model's name and the date, and say whether the geometric decay appears and at what p.
Hint
keep the same interface (__call__(lines, cap) returning a list of lines) so compact.py does not change.
Hard 2: A real extractor for structured state (4 hours)
Replace the patterns in extract with a model that proposes state updates from each turn. Write 20 turns: 10 that state a decision or an identifier in different wording, 5 that merely discuss one ("we could use Redis"), 5 that are noise. (I did not run this.)
Done when: you report recall on the first group and false writes on the second, say how supersession handled a proposal that is not a decision, and say what you would add to the slot schema.
Hint
keep the keying and the gate; only the proposal function changes.
Hard 3: Cache-aware compaction (5 hours)
Add a simple cache model: a request's cost is full price for tokens after the first changed token compared with the previous request, and a tenth of the price for the unchanged prefix. Compare reducing on every step, reducing in batches (only when 20 percent over budget), and reducing at the threshold. (I did not run this.)
Done when: you report cost and accuracy for the three schedules, say which one wins at which price ratio, and say what the stable prefix must contain for the batch schedule to pay off.
Hint
track the previous request's text and compute the common-prefix length in tokens.