Agentic Frameworks & Coordination

6 — Asynchronous Agent Execution

Move long-running agent work into durable jobs with explicit acceptance, progress, cancellation, and completion semantics.

Concept overview

Asynchronous execution separates accepting a request from completing its work. The service returns an operation identifier after durable acceptance, while workers process the operation independently of the user's connection. This is an API contract, not merely a background thread: clients need a reliable way to observe status and retrieve the final result.

The central difficulty is ownership across failures. A queue may deliver a job more than once, a worker can lose its lease, and a client can retry submission after an uncertain response. Use durable operation records, deduplication identities, and idempotent side effects to make these ordinary events safe.

Why it matters in production

Proof of concept

A prototype can use an in-process task for disposable work. Confirm that losing the process and its work is acceptable before choosing that shortcut.

Production

Production persists acceptance before acknowledgement, separates job state from queue state, and gives clients documented polling or notification behavior. Cancellation is cooperative unless the underlying operation supports stronger guarantees.

Enterprise scale

At enterprise scale, fair queues, tenant quotas, backpressure, and bounded worker concurrency protect interactive systems. Measure queue age independently from processing time because autoscaling cannot repair every downstream bottleneck.

Architecture components

Submission API

Validates requests and creates durable operations with stable client-visible identifiers.

Job queue

Buffers accepted work and supports retryable delivery without defining business truth.

Worker pool

Claims leased jobs and performs bounded tool and model operations.

Operation store

Tracks progress, cancellation requests, final artifacts, and failure classifications.

Notification adapter

Publishes completion events or authenticated callbacks without exposing internal job payloads.

The architecture, mapped

Durable asynchronous report execution — The client connection lifetime is independent of the job, and committed operation state determines completion.

sequenceDiagram
 participant U as Analyst
 participant A as Submission API
 participant S as Operation store
 participant Q as Job queue
 participant W as Worker
 U->>A: Request report with submission key
 A->>S: Persist accepted operation
 A-->>U: Operation identifier
 A->>Q: Publish durable job
 Q->>W: Deliver job
 W->>S: Claim lease and checkpoint progress
 W->>S: Commit result and terminal status
 U->>A: Read operation status
 A->>S: Read committed result
 A-->>U: Report link

Runtime flow

1
Accept durably
Validate the request and record its operation identity before returning acceptance.
2
Enqueue reliably
Publish a job through an outbox or equivalent mechanism that closes the database-to-queue gap.
3
Claim ownership
Acquire a renewable lease and verify that the operation remains eligible to run.
4
Execute checkpoints
Save completed stages and progress so a replacement worker can resume safely.
5
Complete atomically
Store the result reference and terminal status before sending a completion notification.
6
Serve status
Let clients poll or reconnect using the operation identifier without creating another job.

Generating a maintenance evidence package

Energy · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A wind-farm operator requests maintenance summaries spanning inspection notes, sensor history, and spare-parts records.

The problem

The report takes longer than an interactive HTTP request and is frequently abandoned when a browser tab closes.

Agent requirement

The assistant must accept the request promptly and produce a recoverable report with visible progress.

Architecture

A submission service creates a durable operation, while leased workers retrieve evidence and assemble a versioned report artifact.

Runtime

The planner submits a site and date range, receives an operation identifier, and continues other work.
Workers checkpoint completed turbine sections; the interface reconnects later and reads the report's actual terminal state.

Failure scenario

If a worker restarts, another worker resumes incomplete sections using the same operation identity and preserves finished artifacts.

Operational impact

Track queue age, completion latency, abandoned operations, duplicate submissions, and worker utilization rather than reporting only API response time.

Architecture tradeoffs

Decision	Benefit	Cost
Immediate acceptance	The interface remains responsive during expensive work.	Clients must understand eventual completion and error reporting.
Durable checkpoints	Worker replacement wastes less completed computation.	Persistence and schema evolution add operational cost.
Shared queues	Capacity can be pooled efficiently.	Fairness controls are necessary to prevent large tenants dominating service.

Failure scenarios & recovery

Notification delivery fails

Detect: Operation is complete but callback attempts remain unsuccessful.

Recover: Retain the result for polling and retry notifications separately.

Two workers overlap after lease expiry

Detect: Ownership version changes while the old worker is still active.

Recover: Fence writes and deduplicate side effects by operation identity.

Common architecture mistakes

Returning acceptance before durable storage

Commit the operation before acknowledging it.

Treating cancellation as guaranteed rollback

Document what can stop and reconcile completed side effects.

Using queue visibility as business state

Keep an authoritative operation record.

When to choose this pattern

Choose it when

Tasks exceed interactive request budgets.
Work must survive client disconnection and worker replacement.
Bursty workloads benefit from controlled buffering.

Avoid it when

The result is required immediately for a synchronous transaction.
Work is trivial and queue overhead dominates latency.
No durable status or operational ownership can be provided.

Architecture checklist

Persist acceptance before responding.
Deduplicate repeated submission keys.
Define polling and result-retention contracts.
Fence stale worker writes.
Separate queue age from processing latency.
Test cancellation during an external tool call.