Preserve business progress across process crashes, long waits, retries, and human approvals without duplicating external effects.
Durable execution records enough workflow history to continue a logical process after workers disappear. It separates durable coordination from activities that call models, tools, or external systems. A restarted worker reconstructs the workflow's recorded decisions instead of treating the entire request as a new task.
Durability is not universal exactly-once execution. An activity can perform a remote effect and fail before its result is recorded. External operations therefore need stable identities, idempotency, status reconciliation, and explicit handling of ambiguous outcomes. Model calls belong in recorded activities so replay does not silently generate a different plan.
A proof of concept can checkpoint a short task, but must identify which waits and side effects survive process termination.
Production adds persisted history, bounded activity retries, durable timers, approval signals, cancellation rules, and operation reconciliation for uncertain external results.
Enterprise scale requires workflow versioning, history retention limits, worker isolation, queue capacity planning, and migrations for long-running processes already in flight.
Advances deterministic business state using persisted events, timers, and activity results.
Retains workflow decisions and completion records needed for reconstruction and audit.
Run model and tool calls outside deterministic coordination and return structured results.
Authenticates external decisions and binds them to the pending workflow revision.
Tracks stable operation keys, submission status, receipts, and reconciliation obligations.
sequenceDiagram
participant W as "Workflow history"
participant A as "Activity worker"
participant S as "Business service"
W->>A: Schedule operation with stable key
A->>S: Submit approved effect
S--xA: Receipt lost after commit
A--xW: Worker stops before result recorded
W->>A: Resume pending activity
A->>S: Query original operation key
S-->>A: Existing effect receipt
A->>W: Record reconciled completionAssign a business-level workflow identifier and deduplicate repeated start requests for the same operation.
Record the intended activity, timeout, retry policy, and stable operation identity before dispatch.
Store structured tool or model output, including partial or uncertain status, before advancing dependent work.
Suspend on a timer or authenticated approval signal without keeping a worker process occupied.
Replay recorded coordination decisions while reusing completed activity results and respecting the workflow's compatible code version.
Resolve ambiguous effects through status queries, then complete or compensate according to the approved business policy.
Logistics · Illustrative enterprise scenario, not a reported customer outcome.
Illustrative architecture: A shipment recovery process waits overnight for carrier acceptance and then requires dispatcher approval of a revised appointment.
Keeping the process in memory loses progress on deployment, while restarting it can duplicate reservations.
Preserve pending ownership, approvals, and reservation receipts across worker restarts and long external waits.
A durable coordinator stores carrier responses and appointment operation keys; workers perform calls and the dispatcher sends a version-bound approval signal.
A worker crashes after reserving a slot; recovery queries the original reservation key before another booking is attempted.
Measure completed workflows, age of pending states, duplicate external effects, recovery time, and unreconciled operations rather than assuming retries imply reliability.
| Decision | Benefit | Cost |
|---|---|---|
| Persisted workflow history | Allows progress to survive process loss. | Adds storage, operational complexity, and retention decisions. |
| Deterministic coordination | Makes reconstruction predictable. | Requires discipline around time, randomness, model calls, and code evolution. |
| Long-lived approval waits | Avoids keeping workers active while humans decide. | Requires expiry, reassignment, and stale-approval policies. |
Detect: Replay compatibility checks or runtime nondeterminism errors identify divergent decisions.
Recover: Route the workflow to a compatible worker version or apply a tested migration strategy.
Detect: The effect ledger shows submission without a definitive receipt.
Recover: Reconcile by operation key and stop blind retries until the business state is known.
Record model output as an activity result and replay that result.
Represent unknown effect state explicitly and query the downstream system.
Bind approvals to evidence and action versions with an expiry policy.