Run it again, resume after a crash, keep secrets out.
Run it again, or replay it
Re-running a task live is not a reproduction:
Method
Result in the example
same task, same seed, run again against the live world
a different email in 85% of runs (the carrier's answer drifts between calls)
exact replay from the trace (recorded decisions, recorded tool results)
the same answer in 100% of runs, with no tool called and nothing sent
Exact replay feeds the loop the model's recorded outputs and the tool's recorded results in order. It works because everything that was random or external was recorded. If the replay ever asks for different input than it saw (the trace holds the original), it stops with a divergence error and says where, instead of silently producing something else. To compare two runs, find the first difference: in the example, a v1 run and a v2 run of the same task differ first at the model's call at step 2 (the arguments of the email), which is exactly where the bug is.
Testing a new prompt on old tool results
A second kind of replay keeps the recorded tool results and swaps in a new model or prompt. The old runs become a regression test that costs no tool calls, sends no email and cannot touch anything real:
What was measured
Wrong date sent
live runs, prompt v1
56%
live runs, prompt v2
14%
200 recorded v1 runs
50%
the same 200 recorded runs replayed with prompt v2
18%
The replayed figure (18%) is close to the live figure for v2 (14%) without a single tool call, which is the point: a prompt change can be checked against yesterday's real traces. The limit is that a new model may ask for a call that was never recorded (a different search, a different argument); the example counts these as misses, and a high miss rate means the replay no longer represents the new behaviour.
The process dies right after a tool ran
Resuming after a crash
A process can die at any moment. The dangerous moment is after a side effect ran but before it was recorded: the email is out, the log does not know. The example kills the process at three points and tries two recoveries:
Restart: start the task again from scratch.
Resume: rebuild the state by folding the saved events, and re-issue the one decision that has no recorded result.
Dies after
restart, no key
resume, no key
restart, with key
resume, with key
get_order
1 email
1 email
1 email
1 email
carrier_status
1 email
1 email
1 email
1 email
send_email
2 emails, different dates
2 emails, same text
1 email
1 email
(Model calls spent in the three rows: restart 4 each time; resume 3, 2 and 1.)
Resuming spends fewer model calls (1 instead of 4 when the crash is after the send), because it keeps the decisions already made.
Restarting can send two emails that disagree, because the model and the world have both changed. This is the worst outcome.
Resuming without a key still sends a duplicate, but at least the same one.
An idempotency key (module 2) makes the send safe either way. Resuming does not remove the need for it: "did my last call take effect?" is not knowable from the trace when the crash comes between the call and the save.
So the rule: record the intent before the effect, record the result after, give every effect a key, and make resume re-issue the unfinished step with the same key.
Secrets and personal data
A trace holds what the model saw and what the tools returned, which means it can hold API keys, passwords, tokens and customer data. In the example one failure message from the carrier includes the request's signing key. Recorded as it stood, the key is in the file; with a redaction step before anything is written, it is replaced by [SECRET], and the replay still works.
Practical rules:
Redact on the way in, not later. A file you must scrub afterwards has already been copied.
Mask by pattern (key formats, emails) and by field name (password, authorization). Keep the mask the same everywhere, so replay still matches: in the example, masked email addresses do not break replay.
Keep traces access-controlled, with a retention period; traces are personal data when the tasks are about people (Level 10).
Decide per field whether you need the value or only a hash or length.