CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 8 of 12 · 1 min read

Answering durability questions in an interview

What happens if a worker dies halfway through a long agent run?

"With durable execution the lease expires, another worker picks the job up, replays the journal to skip completed steps and continues. In the example's simulation journalling cut cost per run from 12.43 to 8.22 at 8 percent crash chance per step. At 20 percent it cut cost from 25.5 to 8.53." Follow-up: "What about side effects?" The crash window between effect and journal write can repeat an effect, so effects take idempotency keys. Journalling alone left 35 duplicates; adding keys brought that to 0.

Why must workflow code be deterministic?

"Replay re-runs the code against the journal; if the code reads the clock or random numbers live, it can take a different path than the original run and disagree with recorded steps. I record such values in the journal on first use and replay them." Follow-up: "What do you do when the workflow definition changes?" Version it and let old runs finish on the old version. The example's engine refuses to resume on a changed structure hash.

Temporal versus a queue with retries?

"A queue with retries restarts the whole job unless I build checkpointing myself. A durable-workflow engine provides the journal, replay, timers and versioning. For short jobs a queue is enough; for long multi-step agents with human waits and expensive steps I want durable execution." Follow-up: "What does it cost?" A deterministic-code constraint, history growth and operational complexity.

How do you handle failure after a payment succeeded?

"A saga: run compensations in reverse (refund, retract) as idempotent steps, log them, and escalate if compensation fails. Better, put human approval before irreversible steps (Level 10 module 5)." Follow-up: "Is compensation guaranteed?" No; some effects cannot be undone, so prefer preventing them.