Resilience, Reliability & Failure Handling

48 — Dead-Letter Queues (DLQ) for Agents

Quarantine repeatedly failing agent jobs with enough context for diagnosis and controlled replay, keeping poison work from blocking healthy processing.

Concept overview

A dead-letter queue receives work that cannot complete under its configured processing policy. It separates poison messages, persistent validation failures, and exhausted retries from the main queue so healthy work can continue. The DLQ is an operational work queue, not a permanent trash bin or a substitute for error classification.

Retain the original business operation identity, payload reference, attempt history, failure reason, and software versions. Replay must preserve idempotency and revalidate current authorization and schema compatibility. Queue-specific semantics matter: redrive thresholds, retention, ordering, and timestamps are provider-dependent and should be tested for the selected queue type.

Why it matters in production

Proof of concept

A prototype can move a deliberately malformed job aside after a bounded number of attempts. Confirm that operators can identify why it failed.

Production

Production defines retryable versus terminal errors, DLQ ownership, alerts, retention, and safe redrive procedures. Sensitive payloads should remain protected even when copied into diagnostic queues.

Enterprise scale

At scale, group failures by signature and deploy targeted fixes before replaying large batches. Uncontrolled redrive can recreate the original outage or overload a newly recovered dependency.

Architecture components

Primary work queue

Delivers accepted agent jobs under documented visibility and redelivery semantics.

Failure classifier

Determines whether another attempt is useful or the job should be quarantined.

Dead-letter store

Retains failed work with operation identity, evidence references, and diagnostic metadata.

Triage console

Supports ownership, grouping, inspection, and disposition of failed jobs.

Redrive controller

Revalidates and requeues selected jobs with rate limits and preserved idempotency.

The architecture, mapped

Poison-job isolation and controlled redrive — Failed work leaves the normal path and returns only after diagnosis, validation, and controlled admission.

flowchart LR
 Q["Primary queue"] --> W["Agent worker"]
 W --> S["Completed operation"]
 W --> C["Failure classifier"]
 C --> Q
 C --> D["Dead-letter queue"]
 D --> T["Triage and repair"]
 T --> V["Replay validation"]
 V --> R["Rate-limited redrive"]
 R --> Q

Runtime flow

1
Attempt bounded processing
Claim a job and apply the workflow's deadline, retry, and idempotency controls.
2
Classify failure
Distinguish transient dependency errors from invalid payloads, denied actions, and exhausted attempts.
3
Quarantine durably
Preserve the job and failure metadata before removing it from ordinary processing.
4
Notify accountable owner
Alert on age, volume, or critical job classes rather than only raw queue size.
5
Repair and validate
Correct code, data, or configuration and test representative failed cases.
6
Redrive carefully
Replay selected jobs under current policy and rate limits, retaining their business operation identities.

Recovering failed invoice ingestion jobs

Supply chain finance · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A distributor ingests supplier invoices through an agent that extracts fields and reconciles purchase-order references.

The problem

A supplier changes its document format, causing the same jobs to fail repeatedly and consume the main worker pool.

Agent requirement

The assistant must isolate affected invoices while continuing to process valid suppliers and preserving recovery evidence.

Architecture

The queue policy sends persistent extraction failures to a protected DLQ with document references, parser versions, and operation keys.

Runtime

Operations staff group failures by supplier format and validate a parser correction on representative samples.
The redrive controller requeues the affected invoices gradually using their original operation identities.

Failure scenario

If an invoice already created a payable before a later stage failed, replay reconciles that payable and resumes incomplete work rather than creating another.

Operational impact

Track oldest failed job, failure signatures, successful redrive, duplicate-effect prevention, and business backlog age.

Architecture tradeoffs

Decision	Benefit	Cost
Separate failure queue	Poison jobs stop consuming ordinary processing capacity.	Operators need ownership and tooling to resolve the backlog.
Rich diagnostics	Root causes can be grouped and reproduced.	Diagnostic data increases storage and privacy responsibilities.
Batch redrive	Large backlogs can recover efficiently.	Excessive replay rate can overload dependencies or repeat systemic failure.

Failure scenarios & recovery

The DLQ has no owner and jobs expire

Detect: Age monitoring shows unresolved work approaching retention limits.

Recover: Assign business ownership and disposition before evidence is lost.

Redrive repeats an already committed command

Detect: Operation ledger shows an existing effect.

Recover: Reuse that result and resume only incomplete stages.

Common architecture mistakes

Treating all failures as retryable until exhaustion

Quarantine deterministic invalid input promptly.

Changing operation keys during replay

Preserve business identity across every delivery.

Redriving the entire queue after a deployment

Validate representative cases and ramp replay gradually.

When to choose this pattern

Choose it when

Queued jobs can fail persistently or contain poison input.
Healthy work should continue during localized failures.
Failed work has a clear triage and replay owner.

Avoid it when

The task is synchronous and no durable job exists.
Failures must be returned immediately with no later recovery.
No process exists to inspect or resolve quarantined work.

Architecture checklist

Define retryable and terminal failure classes.
Preserve operation identity and diagnostic versions.
Assign DLQ ownership and age alerts.
Set retention beyond expected triage time.
Revalidate authorization before replay.
Rate-limit and observe redrive batches.