51 — State Rollbacks

Concept overview

State rollback returns an agent workflow to a previously valid logical state after an incorrect plan, failed validation, or operator intervention. The rollback boundary must distinguish local transactional data, durable workflow history, and effects already committed in external systems. Rewinding the conversation does not cancel a shipment or reverse a payment.

A useful architecture treats recovery as a new, auditable transition. Snapshots provide a starting point; an event history explains changes; compensating commands repair reversible effects. Irreversible actions create a forward-recovery boundary requiring reconciliation or human judgment. Deterministic replay is possible only for recorded inputs and outcomes, not by asking a stochastic model to repeat its earlier reasoning.

Why it matters in production

Proof of concept

A prototype can reset an in-memory state object, but it should already label every tool as read-only, reversible, or irreversible.

Production

Production records checkpoint versions, effect identifiers, and compensation status before exposing a recovery command to operators.

Enterprise scale

Enterprise workflows need retention policies, schema migration of old checkpoints, concurrent-writer fencing, and recovery drills covering several external owners.

Architecture components

Checkpoint store

Persists versioned workflow snapshots with the last committed event position.

Effect ledger

Records requested operations, external receipts, idempotency keys, and compensation eligibility.

Recovery coordinator

Acquires workflow ownership and constructs an ordered recovery plan.

Compensation adapters

Implement domain-specific undo operations with independent retries and verification.

Audit journal

Preserves the original execution and the reason for every recovery decision.

The architecture, mapped

flowchart TD
 A["Failed fulfillment plan"] --> B["Fence active worker"]
 B --> C["Read checkpoint and effect ledger"]
 C --> D["Release stock reservation"]
 C --> E["Shipment already dispatched"]
 D --> F["Create recovered revision"]
 E --> G["Human reconciliation"]
 G --> F

stateDiagram-v2
 [*] --> Running
 Running --> Frozen: Recovery requested
 Frozen --> Compensating: Reversible effects exist
 Frozen --> Reconciliation: Irreversible or unknown effect
 Compensating --> Reconciliation: Compensation unresolved
 Compensating --> Recovered: Invariants verified
 Reconciliation --> Recovered: Business resolution recorded
 Recovered --> Running: New revision resumes

Runtime flow

1. 1

   Stop new work

   Fence the current workflow owner so that a late worker cannot continue committing changes.

2. 2

   Select boundary

   Choose a validated checkpoint and enumerate all later local mutations and external effects.

3. 3

   Classify effects

   Separate reversible reservations from irreversible actions and ambiguous timeouts requiring external lookup.

4. 4

   Compensate in order

   Undo dependent reversible operations in reverse dependency order, recording each result durably.

5. 5

   Restore logical state

   Create a new workflow revision based on the checkpoint after compensation invariants pass.

6. 6

   Resume with evidence

   Continue from the recovery revision or route unresolved effects to an operator with the full ledger.

Recovering a partially executed fulfillment plan

Retail · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

An order agent reserves stock, books a carrier, and requests payment capture across separate services.

The problem

A warehouse exception arrives after the carrier booking but before the workflow reaches its completion checkpoint.

Agent requirement

Recover the order without losing receipts or booking the same carrier twice.

Architecture

A versioned order workflow uses a reservation ledger and explicit carrier-cancellation adapter.

Runtime

• The coordinator freezes the order revision and queries the carrier using its original booking key.
• It cancels a confirmed unused booking, releases inventory, and creates a revised fulfillment task.

Failure scenario

If the carrier already collected the parcel, the workflow enters reconciliation instead of claiming that rollback succeeded.

Operational impact

Track unresolved effects, compensation age, and recovery completion time; the target is a verifiably consistent order, not a clean-looking transcript.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Deleting failed history

Append a recovery revision and preserve the original evidence.

Treating every action as reversible

Document irreversible boundaries and provide forward-recovery procedures.

Replaying model calls to reconstruct facts

Replay stored decisions and tool receipts; explicitly generate a new plan when needed.

When to choose this pattern

Choose it when

• Choose this for multi-step workflows with meaningful recoverable checkpoints.
• Use it when operators must repair a running process without restarting all work.
• Apply it when reversible reservations span several business systems.

Avoid it when

• Avoid rollback semantics for completed irreversible actions.
• Do not add it to a stateless classification endpoint with no durable effects.
• Avoid automatic recovery when compensation rules have not been agreed by system owners.

Architecture checklist

• Classify every tool effect by reversibility.
• Persist checkpoint schema and workflow revision.
• Fence previous owners before recovery.
• Make each compensation independently idempotent.
• Test ambiguous external outcomes during rollback.
• Alert on compensation age and unresolved effects.

Related concepts

• 3 — Agentic State Machines
• 12 — State Persistence
• 16 — Task Interruption & Resumption
• 41 — Idempotent Tool Execution
• 57 — Crash-Safe Memory
• 58 — Partial Success Management

Primary references

• PostgreSQL: Write-Ahead Logging
• AWS Builders Library: Making retries safe with idempotent APIs