Part 5 of 6 · 2 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-goals, run pytest -q and python run_all.py.
Done when: you report finished, false-done and escalated shares for self_report, verified_budget and verified_stall, and say which outcome the budget alone cannot separate.
Hint
goals.run(policy, params, n, seed) returns every column.
Easy 2: Write success criteria (30 minutes)
Take three agent tasks of your own and rewrite each goal as three checkable criteria.
Done when: for each criterion you name the check (a test, a validator, a rubric) and who runs it.
Hint
a checkable criterion names an input and a pass/fail test.
Easy 3: Trace one impossible task (30 minutes)
Run goals.run_task with infeasible=1.0 under verified_budget and verified_stall and print steps and outcome.
Done when: you show how many steps each policy spent and why the second stopped sooner.
Hint
random.Random(seed) and goals.run_task(policy, replace(P, infeasible=1.0), r).
Medium
Medium 1: Choose the threshold (2 hours)
Sweep stall_k from 2 to 15 at 0 percent and 40 percent impossible tasks.
Done when: you plot correct outcomes and cost against the threshold for both and say whether one threshold works for both.
Hint
use replace(P, stall_k=k, infeasible=0.4) with n=10000.
Medium 2: A noisy progress signal (2 hours)
Make the monitor see progress correctly only 90 percent of the time.
Done when: you report how wrong escalations and cost change and what you would change in the threshold.
Hint
flip the progressed flag with probability 0.1 before updating idle.
Medium 3: A graded budget (3 hours)
Scale the budget with the task's value (10, 30, 100 steps) and a cost per step.
Done when: you report the budget that maximises value minus cost for the default mix.
Hint
value - cost_per_step * mean_cost over correct.
Hard
Hard 1: A real agent (5 hours)
Run a real agent on 50 tasks, 10 of them impossible, with self-report and with a verifier. (I did not run this, so your numbers are the ground truth.)
Done when: you report false completions, wrong escalations, steps and tokens, the model and the date, and compare them with the simulation.
Hint
freeze the labels (possible or impossible) before running.
Hard 2: Calibrate the judge (4 hours)
Build a rubric judge for a task, measure its false-approval rate on 100 labelled outputs, and put it into verifier_fp. (I did not run this.)
Done when: you report the measured rate and the false-completion rate the simulation predicts.
Hint
use two labellers or a majority to build the labels.
Hard 3: Detect drift (4 hours)
Add a drift detector that compares each step's action with the goal (embedding similarity or a small model). (I did not run this.)
Done when: you report detection and false-alarm rates and what they do to cost and correctness.
Hint
set a threshold on similarity and count false alarms on on-task steps.