Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-support.
Done when: you can explain why guarded cost more than autonomous at accuracy 0.88.
Hint
guarded escalates most refunds because of the $50 cap.
Easy 2: Change a penalty (30 minutes)
Raise the wrong-action penalty to $100 and re-run.
Done when: you report which policy becomes cheapest and why.
Easy 3: Classify commands (30 minutes)
Add ten commands from your own system to ops.py samples.
Done when: you list any it classifies wrongly and fix one rule.
Medium
Medium 1: Calibrated confidence (3 hours)
Replace the uniform confidence distributions with one calibrated to your own numbers (for example correct answers centred at 0.85).
Done when: you re-plot the threshold sweep and explain how the cliff moved.
Hint
calibration is the whole game: measure it from real traces.
Medium 2: Per-intent thresholds (3 hours)
Give each intent its own threshold and cap.
Done when: you show total cost falls versus a single threshold.
Hint
reversible intents (address change) can tolerate lower thresholds than refunds.
Medium 3: Handoff quality (3 hours)
Add a suggested action and a priority to the packet and test it against a simple human-model that resolves faster with a good packet.
Done when: human minutes per escalation fall in your simulation.
Hard
Hard 1: Shadow mode on real tickets (8 hours)
Run a real classifier on real historical tickets without acting. (I did not run this, so your numbers are the ground truth.)
Done when: you report accuracy, calibration, wrong-action rate and the cost table.
Hard 2: Multi-turn tickets (8 hours)
Extend the agent to ask one clarifying question when two intents are plausible. (I did not run this.)
Done when: you show automation rises without more wrong actions.
Hard 3: Operations sandbox (8 hours)
Put the operations policy in front of a real sandboxed shell in a container. (I did not run this.)
Done when: a destructive command is blocked at both the policy and the container permission level.
Hint
defence in depth: the container's permissions are the last line, not the regex.