Part 3 of 4 · 3 min read
Exercises
Easy
Easy 1: Run it and read the map (30 minutes)
In code/mini-longctx, run pytest -q, python demo.py and python run_all.py (about a minute).
Done when: you report the effective length at 70 and 50 percent from results.json and say in one sentence why the toy's 90 percent threshold is never met.
Hint
E.baseline()["effective"] has the effective lengths by threshold.
Easy 2: Check the harness (30 minutes)
Build a context of 4,000 tokens with the needle at depth 0.25 and print the text around the needle.
Done when: you show that the needle sits at about a quarter of the text and that the extractive reader finds it, and you say what repeat_at_end does to the extractive reader and why.
Hint
haystack.build(4000, 0.25) returns the case; c.needle_at is the character offset.
Easy 3: Move the curve (30 minutes)
Re-run the grid with ToyReader(dip=0.0) and with ToyReader(l0=64000).
Done when: you describe how each change alters the heatmap and say which of the two corresponds to "a model with no position bias" and which to "a model with a longer effective context".
Hint
ToyReader(dip=0.0) removes position bias; l0 is the length at which the length factor halves.
Medium
Medium 1: Paraphrased needle (2 hours)
Change the needle to a paraphrase ("The vault opens with the number 4821") and make ExtractiveReader fail on it. Add a second reader that handles both.
Done when: the old reader misses the paraphrase, the new one finds both, and you say why a pattern reader is the wrong stand-in for a model on paraphrases.
Hint
the reader's regex is in reader.py; keep the answer a four-digit string so scoring still works.
Medium 2: Two needles (3 hours)
Insert two needles at different depths and ask for both ("what are the codes for the vault and the garage?"). Extend the harness to score "both correct".
Done when: on the toy reader both-correct recall equals the product of the single-needle recalls, and you explain why a real model may differ.
Hint
score with all(a in answer for a in answers) and multiply the single-needle recalls for the check.
Medium 3: Your own effective length (3 hours)
Write a threshold sweep: for each threshold from 0.5 to 0.95 report the effective length from a grid.
Done when: you plot the staircase and say what threshold you would choose for a customer-facing assistant and why.
Hint
H.effective_length(g, threshold=t) for each t.
Hard
Hard 1: A real model (4 hours)
Implement a reader that calls your model and run the grid at lengths up to the model's window. (I did not run this, so your numbers are the ground truth.)
Done when: you report the map, the effective length at 90 and 70 percent, the model name and the date, and say whether it has a U shape.
Hint
keep the signature answer(context, question); the harness catches the missing trial argument.
Hard 2: Distractors on a real model (4 hours)
Repeat Hard 1 with 0, 2, 5 and 10 distractors at one length. (I did not run this.)
Done when: you report the recall by distractor count, compare it with the toy's fall, and say which mitigation you would add first.
Hint
haystack.build(..., distractors=n).
Hard 3: Mitigation bake-off (5 hours)
On a real model and a real task of yours, compare reading the full context, retrieve-then-read, repeat-at-end and ends-only placement. (I did not run this.)
Done when: you report accuracy and tokens for each and say which you would ship and under what threshold.
Hint
count retrieved tokens separately from the question tokens.