55 — Distributed Tracing (OpenTelemetry)

Concept overview

Distributed tracing represents a request as timed operations called spans, connected by parent relationships or explicit links. For agents, the important unit is not merely the HTTP call: planning iterations, queued jobs, retrieval, inference, validation, and external effects each contribute to the outcome. OpenTelemetry supplies instrumentation and export conventions while the application defines meaningful workflow boundaries.

Trace context should follow work across HTTP, messaging, and asynchronous execution. A retry needs its own attempt span under the same logical operation; a batch assembled from several requests can use span links. Trace data is operational evidence, not a license to store full prompts, credentials, customer documents, or private reasoning.

Why it matters in production

Proof of concept

A prototype traces one successful turn and verifies that model and tool timing explain its total duration.

Production

Production propagates context through queues, correlates logs, redacts attributes, and defines error classification and sampling.

Enterprise scale

Enterprise scale needs collector capacity, stable semantic naming, controlled cardinality, cross-team ownership, and sampling strategies that retain unusual failures without collecting every payload.

Architecture components

Instrumentation SDK

Creates spans around workflow stages and injects or extracts context at boundaries.

Context carrier

Transfers trace identifiers through request headers or message metadata.

Collector

Receives, filters, batches, and exports telemetry without coupling the agent to one backend.

Trace backend

Stores causal execution records and enables latency and error investigation.

Redaction policy

Allow-lists attributes and protects sensitive prompt or tool data before export.

The architecture, mapped

sequenceDiagram
 participant G as Gateway
 participant A as Agent
 participant Q as Work queue
 participant W as Tool worker
 participant C as Collector
 G->>A: Request with trace context
 A->>Q: Task and context carrier
 Q->>W: Deliver work
 W->>W: Tool attempt span
 W-->>A: Result and operation identifier
 A-->>G: Validated answer
 A-->>C: Agent spans
 W-->>C: Linked worker spans

Runtime flow

1. 1

   Start logical request

   Create a root span carrying a stable task type, tenant classification, and workflow revision without raw personal data.

2. 2

   Propagate execution context

   Inject context when calling tools or publishing work; extract it in the receiving process.

3. 3

   Create semantic child spans

   Separate retrieval, provider queueing, inference, validation, and side-effect commit timings.

4. 4

   Record attempt outcomes

   Attach model revision, token usage, retry reason, and structured error category to the appropriate attempt.

5. 5

   Export safely

   Batch telemetry through a collector with bounded buffers and backpressure behavior that cannot block business work indefinitely.

6. 6

   Investigate causal paths

   Correlate a failed workflow with logs and metrics while distinguishing parallel work from critical-path latency.

Finding the slow step in a maintenance agent

Manufacturing · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A plant assistant reads equipment alarms, retrieves manuals, and asks a maintenance API for recent work orders.

The problem

Operators report occasional slow answers, but aggregate model latency appears normal.

Agent requirement

Locate the causal bottleneck without exporting proprietary manuals or machine credentials.

Architecture

Instrumentation wraps retrieval and work-order calls, propagates context through a job queue, and exports only approved identifiers.

Runtime

• A slow interaction produces separate queue, retrieval, and inference spans.
• The trace reveals repeated work-order timeouts before the model starts generating.

Failure scenario

If the collector becomes unavailable, bounded local buffering drops telemetry according to policy while the assistant continues serving.

Operational impact

Track trace completeness, critical-path time, and dropped telemetry; the value is actionable diagnosis rather than a larger collection of disconnected logs.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Recording complete prompts by default

Use an attribute allow-list and separate governed evidence storage.

Treating retries as one opaque span

Record each attempt and its reason under a logical operation.

Making telemetry export synchronous

Use bounded asynchronous export with explicit loss metrics.

When to choose this pattern

Choose it when

• Use for workflows crossing services or asynchronous queues.
• Use when latency and cost cannot be explained by provider metrics alone.
• Use to correlate tool effects with the agent request that caused them.

Avoid it when

• Do not treat traces as an authoritative transaction ledger.
• Avoid storing secrets or unrestricted model payloads in span attributes.
• Avoid tracing every token as an individual span at high throughput.

Architecture checklist

• Propagate context through every transport.
• Name spans by stable operations.
• Separate attempts from logical requests.
• Redact attributes before export.
• Measure collector drops and queue depth.
• Verify one complete failure trace end to end.

Related concepts

• 1 — Multi-Agent Orchestration
• 6 — Asynchronous Agent Execution
• 30 — Non-Repudiation Logs
• 49 — Timeout Policies
• 60 — Chaos Engineering for AI
• 94 — Token Cost Allocation Tracking

Primary references

• OpenTelemetry: Traces