49 — Timeout Policies

Concept overview

Timeout policies bound how long a caller waits for an operation; deadlines express when the overall task must finish. Use both deliberately. A per-call timeout that resets at every layer can exceed the user's total budget, especially when retries, queueing, and nested agent calls accumulate.

Distinguish connection establishment, response headers, idle streaming, and total execution time. A timeout does not prove that the remote operation stopped or failed. For commands, the result may be unknown even after the caller gives up, so preserve operation identity and reconcile before retrying. Cancellation is cooperative unless the remote system offers stronger guarantees.

Why it matters in production

Proof of concept

A prototype should set explicit model and tool timeouts and record actual latency distributions. Include slow connections and long silent streams in tests.

Production

Production propagates a parent deadline, allocates remaining time to each stage, and reserves finalization capacity. Retry decisions must account for whether another attempt can finish usefully before the deadline.

Enterprise scale

At scale, tune by task class and dependency percentiles, including network and queue delay. Watch false timeouts and abandoned downstream work because overly aggressive limits can amplify load through retries.

Architecture components

Deadline context

Carries the task's remaining time through workers, tools, and delegated agents.

Stage budget allocator

Divides available time among required work, retries, and finalization.

Timeout adapter

Applies connection, idle, and total-call limits supported by each client library.

Cancellation coordinator

Stops optional work and propagates cooperative cancellation when the task expires.

Outcome reconciler

Determines whether timed-out commands actually committed before further action.

The architecture, mapped

flowchart LR
 D["Parent deadline"] --> Q["Queue wait"]
 Q --> R["Remaining-time check"]
 R --> M["Bounded model call"]
 M --> T["Bounded tool call"]
 T --> F["Reserved finalization"]
 R --> X["Deadline exhausted"]
 M --> X
 T --> X
 X --> C["Cancel and reconcile"]
 C --> O["Explicit terminal status"]

sequenceDiagram
 participant A as Agent
 participant T as Remote command service
 participant R as Reconciler
 A->>T: Command with stable operation key
 T->>T: Commit effect
 Note over A: Caller deadline expires
 A->>R: Record uncertain outcome
 R->>T: Query original operation key
 T-->>R: Confirm committed result
 R-->>A: Update durable operation status

Runtime flow

1. 1

   Set task deadline

   Derive the allowed completion window from the user-facing service contract.

2. 2

   Charge elapsed time

   Include queue wait, prior stages, and delegation overhead in the remaining budget.

3. 3

   Allocate call limit

   Choose a timeout no longer than the useful remaining task window after reserves.

4. 4

   Execute and observe

   Distinguish connection, idle, and total timeout reasons in telemetry.

5. 5

   Handle expiry

   Stop new work, cancel where supported, and classify commands as uncertain until reconciled.

6. 6

   Finalize honestly

   Return complete, partial, deferred, or failed status with the known operation outcome.

Handling an uncertain reservation hold

Travel management · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A corporate travel assistant checks availability and places a temporary reservation hold after user confirmation.

The problem

The booking service can accept a hold while its response is delayed beyond the client timeout.

Agent requirement

The assistant must respect the interactive deadline without placing duplicate holds or falsely reporting failure.

Architecture

A parent deadline covers search, confirmation, and hold creation; the command uses a stable operation key and reconciliation endpoint.

Runtime

• The assistant spends part of its budget on availability, then allocates only the remaining safe window to hold creation.
• If the call times out, the interface reports a pending outcome while reconciliation checks the original operation key.

Failure scenario

A confirmed existing hold is returned to the user; no new hold is created merely because the first response was late.

Operational impact

Track timeout reason, false-timeout rate, downstream work after cancellation, and time to reconcile uncertain holds.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Assuming timeout means no side effect occurred

Reconcile commands using stable operation identity.

Ignoring queue delay in the budget

Charge time from accepted request to final delivery.

Using one timeout for all task classes

Tune against dependency behavior and user expectations.

When to choose this pattern

Choose it when

• Requests depend on remote or variable-latency services.
• Users need predictable completion or explicit deferred status.
• Retries and nested calls could exceed total budgets.

Avoid it when

• The operation is intentionally unbounded with no useful deadline.
• A timeout would be interpreted as guaranteed remote cancellation.
• The system cannot represent uncertain command outcomes.

Architecture checklist

• Define end-to-end service deadlines.
• Include queue and connection time.
• Reserve finalization and reconciliation capacity.
• Propagate remaining time to children.
• Test cancellation and late responses.
• Separate timeout reasons in telemetry.

Related concepts

• 6 — Asynchronous Agent Execution
• 16 — Task Interruption & Resumption
• 41 — Idempotent Tool Execution
• 42 — Circuit Breakers (Resilience4j)
• 50 — Exponential Backoff with Jitter

Primary references

• AWS: timeouts, retries, and backoff with jitter
• Amazon EC2: ensuring idempotency