Resilience, Reliability & Failure Handling

59 — Cascade Failure Prevention

Contain overload and dependency failures so one slow tool or model does not consume the capacity of an entire agent platform.

Concept overview

Cascade prevention limits the propagation of failure across queues, retries, threads, connection pools, and downstream services. Agent workflows amplify this risk through fan-out: one user request may create several model calls and many tool operations. When a dependency slows, in-flight work grows; retries then add load precisely when less capacity is available.

The control system needs several cooperating boundaries. Deadlines cap waiting, bulkheads isolate capacity, circuit breakers stop repeated calls to failing dependencies, admission control bounds demand, and graceful degradation reduces optional work. Each boundary must preserve enough information to distinguish rejected work from work whose external outcome is uncertain.

Why it matters in production

Proof of concept

A prototype should count downstream calls per task and deliberately slow a tool to reveal unbounded fan-out.

Production

Production introduces end-to-end deadlines, retry budgets, concurrency isolation, and overload responses before queues saturate.

Enterprise scale

At scale, model regional failures, synchronize neither retries nor recovery probes, and reserve capacity for critical tenants and administrative recovery paths.

Architecture components

Admission controller

Rejects or defers work before local resources are exhausted.

Bulkhead pools

Separate concurrency budgets by dependency or workload priority.

Circuit breaker

Uses recent outcomes to stop calls and permit limited recovery probes.

Deadline propagator

Carries remaining time through nested operations and cancels expired work.

Retry budget

Caps additional attempts across the entire logical request.

The architecture, mapped

Containing a slow dependency — Independent capacity prevents a billing slowdown from consuming unrelated search and recovery resources.

flowchart LR
 U["Incoming agent tasks"] --> A["Admission limit"]
 A --> C["Critical task pool"]
 A --> O["Optional enrichment pool"]
 C --> B["Billing breaker"]
 B --> T["Slow billing API"]
 B --> D["Defer billing task"]
 O --> S["Search service"]
 T --> H["Bounded recovery probes"]
 H --> B

Controlled dependency recovery — Limited probes and gradual traffic restoration avoid a synchronized retry surge.

stateDiagram-v2
 [*] --> Closed
 Closed --> Open: Failure threshold reached
 Open --> HalfOpen: Probe delay elapsed
 HalfOpen --> Open: Probe fails
 HalfOpen --> Recovering: Probe succeeds
 Recovering --> Closed: Gradual traffic ramp is healthy
 Recovering --> Open: Failures return

Runtime flow

1
Estimate fan-out
Calculate whether the requested plan fits remaining time and downstream capacity.
2
Acquire isolated capacity
Reserve a slot in the relevant dependency pool rather than a shared unlimited executor.
3
Check breaker state
Fail fast or choose an approved fallback when a dependency is known to be unhealthy.
4
Execute within deadline
Propagate cancellation and avoid starting work that cannot finish within the request budget.
5
Retry within budget
Apply bounded backoff with jitter only for eligible failures and known-safe effects.
6
Recover gradually
Allow limited probes and ramp traffic after sustained evidence of recovery.

Storm-driven service requests

Energy · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A utility assistant handles outage reports, account inquiries, and restoration explanations during severe weather.

The problem

A slow account API causes retries and consumes worker slots needed for outage reporting.

Agent requirement

Keep critical outage intake available while nonessential enrichment degrades.

Architecture

Separate pools protect outage intake, and the account adapter uses a breaker with a shared retry budget.

Runtime

The account pool fills and new account enrichment is deferred.
Outage tasks continue through their reserved capacity while a small probe stream checks account recovery.

Failure scenario

If the breaker closes too aggressively, fresh failures reopen it and the recovery ramp pauses.

Operational impact

Track rejected optional work, protected-task latency, concurrency saturation, and retry amplification; a healthy platform may intentionally serve fewer noncritical features.

Architecture tradeoffs

Decision	Benefit	Cost
Early load shedding	Preserves useful capacity during overload.	Some requests are rejected despite a running process.
Dependency bulkheads	Limit failure propagation.	Require capacity allocation and can reduce sharing efficiency.
Circuit breakers	Reduce pressure on failing services.	Bad thresholds can delay recovery or oscillate.

Failure scenarios & recovery

Retries multiply across layers

Detect: Downstream attempts greatly exceed original request volume.

Recover: Assign one retry owner and enforce a request-wide budget.

Synchronized recovery stampede

Detect: Many replicas probe or reconnect simultaneously.

Recover: Add jitter, cap probe concurrency, and ramp accepted traffic.

Common architecture mistakes

Using timeouts without cancellation

Stop expired downstream work where supported.

Sharing one executor for all tools

Separate critical and optional dependency capacity.

Treating every failure as retryable

Classify errors and reconcile ambiguous effects before replay.

When to choose this pattern

Choose it when

Use for fan-out workflows and shared dependencies.
Use where traffic surges threaten critical operations.
Use when retries contribute materially to resource use.

Avoid it when

Avoid breaker complexity around trivial local functions.
Do not fail open across authorization boundaries.
Do not shed critical work without a documented deferred path.

Architecture checklist

Set a workflow-wide deadline.
Measure downstream calls per original request.
Isolate critical resource pools.
Bound retry attempts and duration.
Jitter recovery probes.
Load-test degradation and recovery, not only steady state.