Part 7 of 12 · 1 min read
How to build it so it holds up
- Journal every expensive or non-repeatable step before moving on; store results, not just "done".
- Give each effect an idempotency key derived from run id and step name, and use it downstream.
- Keep orchestration code deterministic; put LLM calls, I/O, time and randomness inside recorded steps or
Ctx-style recorded values. - Version workflows; refuse to resume a journal against a different structure, or run old versions to completion.
- Model waits as journalled suspensions, with timeouts and escalation (Level 10 module 9).
- Plan compensations for effects that matter; make them idempotent; log them.
- Heartbeat and lease (module 2) so a dead worker's run is picked up by another.
- Cap total run time, steps and cost even for durable runs (Level 9 budgets).
- Test by killing workers at random positions in staging (chaos testing).
- Choose build or buy deliberately: a journal table plus a queue covers small cases; durable-workflow engines (Temporal and similar) or framework checkpointers cover the rest.