57 — Crash-Safe Memory

Concept overview

Crash-safe memory preserves the workflow facts needed for correct recovery after volatile process state disappears. It includes accepted inputs, state revisions, pending commands, tool receipts, and completion markers. Saving conversation text alone cannot establish whether an external operation happened before a crash.

The critical design question is the commit boundary. Durable state must precede acknowledgment, and external effects need idempotency or reconciliation when they cannot share a transaction with workflow storage. Write-ahead logging protects a database transaction under its configured durability assumptions; it does not automatically make the surrounding distributed workflow exactly once.

Why it matters in production

Proof of concept

A prototype should survive a forced process kill between two tool calls without losing the accepted task.

Production

Production persists revisions and effect identifiers, coordinates queue acknowledgment with durable commits, and tests duplicate delivery.

Enterprise scale

Enterprise scale defines recovery point and recovery time objectives, replication behavior, backup restoration, and schema compatibility for old unfinished workflows.

Architecture components

Durable state database

Stores workflow revisions and committed task progress.

Command outbox

Records intended external work in the same transaction as its originating state change.

Receipt ledger

Associates tool results with stable operation identifiers.

Recovery worker

Acquires a lease and reconstructs pending work after failure.

Snapshot archive

Provides recoverable history beyond a single running database instance.

The architecture, mapped

sequenceDiagram
 participant W as Agent worker
 participant D as Durable database
 participant T as Tool service
 participant R as Recovery worker
 W->>D: Commit state and command intent
 W->>T: Execute with operation key
 T-->>W: Receipt
 W->>D: Commit receipt and progress
 R->>D: Read pending commands after crash
 R->>T: Reconcile operation key
 T-->>R: Existing outcome

stateDiagram-v2
 [*] --> IntentCommitted
 IntentCommitted --> EffectSubmitted
 EffectSubmitted --> ReceiptCommitted: Result persisted
 EffectSubmitted --> OutcomeUnknown: Worker crashes
 OutcomeUnknown --> Reconciling: Recovery lease acquired
 Reconciling --> ReceiptCommitted: Existing effect found
 Reconciling --> IntentCommitted: Safe retry established
 Reconciling --> HumanReview: External outcome unresolved
 ReceiptCommitted --> [*]

Runtime flow

1. 1

   Accept durably

   Commit the task identifier and initial state before acknowledging receipt to the caller.

2. 2

   Record intent

   Write the next state and tool command together using a transaction or equivalent atomic boundary.

3. 3

   Execute command

   Deliver the command with a stable idempotency key and retain its external receipt.

4. 4

   Commit outcome

   Persist validated results and mark the command complete before advancing the workflow.

5. 5

   Recover ownership

   A replacement worker acquires a fenced lease and reads the last committed revision.

6. 6

   Resolve uncertainty

   Reconcile incomplete commands against external receipts before deciding whether execution may repeat.

Surviving a crash during account-maintenance review

Banking operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

An internal agent verifies supporting documents and submits approved account-maintenance requests.

The problem

A worker can crash after submission but before recording the response.

Agent requirement

Recover the review without submitting the maintenance request twice.

Architecture

The workflow database stores command intent, a stable case key, and the maintenance-system receipt.

Runtime

• The replacement worker finds a pending submission for the case.
• It queries the maintenance system by the original key and records the existing receipt.

Failure scenario

If the external system cannot identify the submission, the case pauses for reconciliation instead of assuming failure.

Operational impact

Measure recovered task completeness, duplicate-effect incidents, and reconciliation age; memory durability is useful only when business effects remain coherent.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Persisting only model messages

Store workflow state, tool intent, and effect receipts.

Assuming database durability covers APIs

Design idempotency and reconciliation across the external boundary.

Never testing abrupt termination

Inject kills at every commit boundary and inspect recovered invariants.

When to choose this pattern

Choose it when

• Use for long-running or high-value workflows.
• Use with at-least-once queues and replaceable workers.
• Use when accepted work must survive machine loss.

Avoid it when

• Avoid durable orchestration for disposable stateless inference.
• Do not promise zero data loss without storage guarantees.
• Avoid automatic replay of ambiguous non-idempotent commands.

Architecture checklist

• Define the durable acknowledgment boundary.
• Persist command identity before execution.
• Fence recovered workers.
• Test crashes between effect and receipt commit.
• Document replication and backup recovery objectives.
• Exercise restoration of unfinished workflow schemas.

Related concepts

• 12 — State Persistence
• 16 — Task Interruption & Resumption
• 38 — Data Lineage Tracking
• 41 — Idempotent Tool Execution
• 51 — State Rollbacks
• 58 — Partial Success Management

Primary references

• PostgreSQL: Write-Ahead Logging
• AWS Builders Library: Making retries safe with idempotent APIs