CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 2 of 12 · 3 min read

How a run survives the process that was running it

A save point for agent runs: what gets written down, and what happens on restart.

Think of a save point in a game. If the game crashes, you restart from the last save, not from the beginning. Durable execution gives an agent run the same thing.

In plain words: after every finished step, the worker writes down what the step was and what it returned. If the worker dies, a new worker reads that list, skips everything on it, and carries on from the first step that is missing.

A worked example. The example agent has seven steps. Four are model calls with a cost (plan 1, research 3, draft 2, write_report 2, so 8 in total) and three touch the outside world: charge a card, send an email, update the CRM.

1 plan          (model, cost 1)
2 research      (model, cost 3)
3 draft         (model, cost 2)
4 charge_card   (effect)
5 write_report  (model, cost 2)
6 send_email    (effect)
7 update_crm    (effect)

Suppose the worker is killed just after step 5. Without a journal, the new worker starts at step 1: it spends another 8 on model calls, and it charges the card a second time. With a journal, the new worker reads five records, skips steps 1 to 5 without running them, and starts at step 6. Extra spend: 0. Extra charges: 0. That is the whole idea; everything else in this module is about the corner cases.

Durable execution

Running code so that a run's progress survives the death of the process running it.

Example: a worker killed by a deploy at step 5 of 7; a fresh worker on another machine finishes steps 6 and 7.

Journal (event history)

An ordered list of completed steps, each with its result, written as the run goes. It is the save file.

Example: after step 3 the journal reads [plan -> "plan-out", research -> "research-out", draft -> "draft-out"]. Storing the result matters: the next step needs the draft, not the fact that drafting happened.

The key lines. The whole mechanism is one loop. Here it is from code/mini-durable/durable.py, trimmed to the lines that matter:

# from code/mini-durable/durable.py
for i, s in enumerate(steps):
    if use_journal and i < len(journal.records):   # this step is already in the journal
        ...
        continue                                   # replay: skip it, reuse the recorded result
    if s.kind == "wait":
        ...                                        # a pause for a human (section 4)
    elif s.kind == "effect":
        result = world.do_effect(s.name, run_id, idem)
    else:
        world.spent += s.cost; result = f"{s.name}-out"   # a model call: pay for it
    if use_journal: journal.records.append({"name": s.name, "result": result})

Read it as: for each step, if the journal already has it, skip; otherwise run it and write it down. A restart simply calls this loop again with the same journal.

Replay

Running the workflow code again from the top with the journal beside it, so that every recorded step is skipped and only unfinished steps execute.

Example: on restart after step 5, the loop above hits continue five times and runs for the first time at step 6.