41 — Idempotent Tool Execution

Concept overview

Idempotent execution lets a caller safely repeat an operation without repeating its intended business effect. Agent systems need this because model loops, worker recovery, queue redelivery, and network timeouts can all create duplicate attempts. The idempotency identity belongs to the business operation, not to an individual HTTP attempt.

Persist the key with a canonical request fingerprint and the operation result. A repeated key with different parameters must be rejected rather than silently treated as the original request. Exactly-once delivery is generally not the useful promise; design for repeated delivery with controlled effects and explicit reconciliation of uncertain outcomes.

Why it matters in production

Proof of concept

A prototype can assign one stable key to each simulated command and test duplicate submission. Do not generate a fresh key inside retry logic.

Production

Production atomically claims operation identity, stores status and results, and coordinates with the actual side effect. If the remote system cannot participate, use its idempotency support or a reconciliation protocol.

Enterprise scale

At scale, choose key scope and retention from the maximum replay horizon. Expiring keys too early allows delayed duplicates, while retaining them indefinitely increases storage and data-management cost.

Architecture components

Operation identity

Names the intended business action across retries, workers, and model turns.

Request fingerprint

Canonicalizes meaningful parameters and detects conflicting reuse of a key.

Idempotency ledger

Atomically tracks pending, completed, failed, and uncertain operation outcomes.

Tool adapter

Passes stable identity to supported downstream APIs and normalizes responses.

Reconciler

Resolves ambiguous outcomes using authoritative business records before another attempt.

The architecture, mapped

sequenceDiagram
 participant A as Agent worker
 participant L as Idempotency ledger
 participant T as Order service
 A->>L: Claim operation key and request hash
 L-->>A: New operation accepted
 A->>T: Create replacement with stable key
 T-->>A: Replacement identifier
 A->>L: Store completed result
 A->>L: Retry same key after lost response
 L-->>A: Return existing replacement result

stateDiagram-v2
 [*] --> Unseen
 Unseen --> Pending: Atomically claim key and fingerprint
 Pending --> Completed: Authoritative effect confirmed
 Pending --> Uncertain: Response lost
 Uncertain --> Completed: Reconciliation finds effect
 Uncertain --> Pending: Safe retry with same key
 Completed --> Completed: Same key and same intent
 Pending --> Conflict: Same key with changed parameters
 Completed --> Conflict: Same key with changed parameters
 Conflict --> [*]

Runtime flow

1. 1

   Define operation

   Derive or create a stable key for the specific business intent and tenant scope.

2. 2

   Validate fingerprint

   Compare canonical parameters with any existing entry for that key.

3. 3

   Claim execution

   Atomically reserve the operation or return the already recorded result.

4. 4

   Perform effect

   Invoke the downstream command with matching idempotency support where available.

5. 5

   Commit result

   Store the authoritative outcome and response reference before acknowledging completion.

6. 6

   Reconcile uncertainty

   If confirmation is lost, query operation status and reuse the same key for any permitted retry.

Avoiding duplicate replacement shipments

Retail fulfillment · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A retailer's assistant creates replacement shipments after staff approve damaged-product cases.

The problem

The shipping API can accept a request and then lose the response, causing the agent to issue a second shipment.

Agent requirement

The assistant must recover from uncertainty without sending duplicate goods.

Architecture

A replacement operation key binds the claim, order line, approved quantity, and tenant to an idempotency ledger and downstream request.

Runtime

• The worker claims the key, submits the replacement, and stores the shipping system's identifier.
• A retry from another worker returns the existing result or reconciles the pending operation before attempting delivery again.

Failure scenario

If the same key arrives with a changed address or quantity, the gateway rejects the conflict and requires a new explicitly authorized business operation.

Operational impact

Measure duplicate effects, unresolved operations, conflicting-key reuse, and reconciliation latency. The ledger alone cannot guarantee safety if the external command bypasses its protocol.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Generating a new key for every attempt

Create the key once per business intent.

Deduplicating only in process memory

Persist identity across restarts and workers.

Treating identical text as identical intent

Use explicit operation identity and canonical parameters.

When to choose this pattern

Choose it when

• Tools create or modify business records.
• Retries, replay, or duplicate delivery are possible.
• External effects can expose stable identity or reconciliation.

Avoid it when

• The operation is a pure read with no side effects.
• A repeated action is intentionally a new business request.
• The downstream effect cannot be identified or reconciled safely.

Architecture checklist

• Define key scope and replay lifetime.
• Reject conflicting parameter reuse.
• Claim operations atomically.
• Preserve keys across every retry layer.
• Test crashes around effect confirmation.
• Monitor pending and uncertain operations.

Related concepts

• 6 — Asynchronous Agent Execution
• 11 — Tool Invocation & Function Calling
• 12 — State Persistence
• 48 — Dead-Letter Queues (DLQ) for Agents
• 50 — Exponential Backoff with Jitter

Primary references

• Amazon EC2: ensuring idempotency