Resilience, Reliability & Failure Handling

45 — Graceful Degradation

Preserve the most valuable safe subset of service during dependency failure, making reduced capability and evidence freshness explicit to users.

Concept overview

Graceful degradation deliberately reduces functionality when full service cannot be delivered. An agent might provide verified cached information, omit an optional enrichment, queue a long task, or return a structured partial result. The degraded mode must have its own correctness contract rather than merely returning whatever content happens to be available.

Classify dependencies and capabilities by business consequence. Losing a recommendation model may permit a search-only experience; losing authorization cannot justify unrestricted access. Define which facts may be stale, which actions are disabled, and how the interface communicates limitations without falsely claiming complete success.

Why it matters in production

Proof of concept

A prototype can disable one dependency at a time and identify which useful functions remain. Write down the minimum acceptable response for each failure.

Production

Production implements explicit capability states, fallback ordering, freshness limits, and observability for degraded responses. Keep degraded modes exercised so they do not become untested emergency code.

Enterprise scale

At scale, avoid shifting all traffic onto an undersized fallback. Capacity reservations, admission control, and recovery hysteresis prevent degradation from triggering another outage.

Architecture components

Capability map

Relates user-visible functions to mandatory and optional dependencies.

Health and policy evaluator

Determines which degraded modes are currently safe and permitted.

Fallback provider

Supplies validated cached data, simpler computation, alternative models, or queued work.

Response contract

Marks missing capabilities, evidence freshness, and incomplete outcomes explicitly.

Recovery controller

Restores normal service gradually and observes fallback capacity and errors.

The architecture, mapped

Capability-aware degradation — Each reduced mode preserves mandatory controls and tells the user which capabilities are unavailable.

flowchart LR
 Q["Customer request"] --> C["Capability and health evaluator"]
 C --> F["Full assisted workflow"]
 C --> S["Verified search-only mode"]
 C --> K["Freshness-bounded cache"]
 C --> J["Deferred job"]
 F --> R["Explicit response contract"]
 S --> R
 K --> R
 J --> R
 P["Mandatory security policy"] --> C

Runtime flow

1
Assess dependency state
Detect timeout, open circuit, unavailable data, or exhausted capacity using trusted signals.
2
Select permitted mode
Consult the capability map and preserve mandatory security and correctness constraints.
3
Bound fallback demand
Apply admission and concurrency limits before moving work to alternate capacity.
4
Produce useful subset
Return only results that meet the degraded contract and freshness requirements.
5
Explain limitations
Identify missing analysis or disabled actions without presenting partial service as complete.
6
Recover gradually
Probe primary health and restore capabilities with hysteresis and outcome monitoring.

Keeping store support useful during model outage

Retail operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A retailer's store assistant answers operating-procedure questions and explains current stock exceptions.

The problem

When the language-model provider is unavailable, the entire support interface becomes unusable even though approved procedures remain searchable.

Agent requirement

The assistant must preserve access to verified information while disabling unsupported analysis and stock-changing actions.

Architecture

A capability map selects search-only mode for procedure questions and queues analytical requests that require unavailable inference.

Runtime

A store manager asks for a returns procedure; the interface shows relevant approved source passages with their version dates.
A complex stock question receives a deferred-operation identifier and a clear description of the unavailable analysis.

Failure scenario

If cached inventory exceeds its freshness limit, the system withholds availability claims rather than showing stale stock as current.

Operational impact

Track successful degraded tasks, fallback saturation, stale-data rejection, and time to restore full capability. Availability is measured against each mode's promised behavior.

Architecture tradeoffs

Decision	Benefit	Cost
Partial service	Users can complete some work during an outage.	Additional modes increase implementation and testing effort.
Cached evidence	Responses remain fast when origins fail.	Freshness limits restrict which claims remain valid.
Deferred work	Important tasks can survive temporary capacity loss.	Users need durable status and realistic completion expectations.

Failure scenarios & recovery

Fallback capacity overloads after primary failure

Detect: Queue age and rejection rate rise sharply in the alternate path.

Recover: Shed optional work and enforce reserved capacity limits.

The interface labels a partial result complete

Detect: Contract checks find missing mandatory sections without status markers.

Recover: Correct the response state and block misleading publication.

Common architecture mistakes

Degrading authorization to preserve availability

Keep mandatory access controls fail-safe.

Using stale data without disclosure

Enforce freshness bounds and show source age.

Never testing fallback modes

Exercise them in routine validation and controlled failure tests.

When to choose this pattern

Choose it when

Useful safe capabilities survive some dependency failures.
Users can understand and act on partial results.
Fallback capacity and freshness can be bounded.

Avoid it when

A partial result would be misleading or unsafe.
Mandatory authorization or integrity controls are unavailable.
The fallback has no validated correctness contract.

Architecture checklist

Classify mandatory and optional dependencies.
Define each degraded mode's contract.
Set evidence freshness limits.
Reserve and limit fallback capacity.
Expose partial and deferred status clearly.
Test gradual restoration and mode switching.