54 — Memory Leak Mitigation in Long-Running Agents

Concept overview

Memory leak mitigation addresses resources that remain reachable or allocated after their useful lifetime. Agent runtimes are especially exposed because conversation histories, tool transcripts, callback closures, streaming buffers, and background tasks can survive many turns. GPU memory adds a separate distinction between live tensors, cached allocator blocks, and key-value cache retained for active sequences.

A rising resident-set size is a symptom rather than proof of a leak. Capacity growth, allocator fragmentation, unbounded caches, native-library allocations, and true lost ownership require different remedies. The architectural goal is explicit ownership, bounded cardinality, and observable release at session, request, and worker boundaries.

Why it matters in production

Proof of concept

A prototype should run repeated conversations in one process and measure whether retained objects return toward a stable baseline.

Production

Production introduces session expiration, bounded caches, stream cancellation, task ownership, and memory-aware admission limits.

Enterprise scale

Enterprise operation adds long-duration soak tests, per-tenant quotas, heap sampling, and controlled worker recycling as a containment mechanism while root causes are repaired.

Architecture components

Session store

Keeps durable history outside the worker and loads only the required turn window.

Resource scope manager

Closes clients, cancels child tasks, and releases buffers when a request ends.

Bounded cache

Enforces byte and entry limits with eviction and expiry rather than unlimited memoization.

Memory profiler

Separates heap objects, native allocation, process residency, and accelerator allocation.

Admission controller

Limits concurrent contexts using measured memory cost, not request count alone.

The architecture, mapped

flowchart LR
 H["Durable session history"] --> W["Bounded turn window"]
 W --> S["Request resource scope"]
 S --> B["Tool buffers and child tasks"]
 B --> C["Completion or cancellation"]
 C --> X["Close handles and release references"]
 P["Heap and GPU telemetry"] --> Q["Drain oversized worker"]

Runtime flow

1. 1

   Assign ownership

   Attach every tool stream, task, and temporary artifact to an explicit request or session scope.

2. 2

   Load a bounded working set

   Retrieve current context while keeping full history in durable external storage.

3. 3

   Observe growth

   Record memory before and after representative turns and correlate with session and token counts.

4. 4

   Close the scope

   Cancel unfinished tasks, release stream readers, and remove callbacks when the caller disconnects or the task ends.

5. 5

   Evict and expire

   Remove idle sessions and cache entries under byte budgets and tenant limits.

6. 6

   Contain abnormal workers

   Drain and replace a worker exceeding its memory envelope while retaining diagnostic evidence.

A dispatch assistant that stays active across shifts

Logistics · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

Dispatchers keep long conversations open while checking routes, driver status, and delivery exceptions.

The problem

Workers retain every tool response and disconnected stream listener, so memory grows across shift changes.

Agent requirement

Support ongoing conversations while bounding each worker footprint and preserving cancellation behavior.

Architecture

The service stores full transcripts in a database, loads a compact working window, and scopes tracking streams to active requests.

Runtime

• A dispatcher opens a vehicle-status stream and later closes the browser.
• Disconnect propagation cancels the stream, removes callbacks, and expires the idle session cache.

Failure scenario

If a native map library continues growing outside the managed heap, the worker drains while profiles identify the native allocation path.

Operational impact

Measure memory per active session, retained bytes after cancellation, and out-of-memory termination rate; worker restarts are containment rather than evidence of a repaired leak.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Diagnosing only managed heap size

Measure native and accelerator allocations alongside application objects.

Capping cache entries but not bytes

Apply byte budgets because transcript and embedding sizes vary.

Clearing all memory on a timer

Use ownership and expiry policies that preserve active requests.

When to choose this pattern

Choose it when

• Use for persistent agents, streaming tools, and long-lived workers.
• Use when memory correlates with cumulative turns rather than current load.
• Use before increasing concurrency for long-context inference.

Avoid it when

• Do not treat expected allocator caching as a confirmed leak.
• Avoid indiscriminate eviction of active sequence state.
• Do not rely exclusively on larger containers to mask unbounded growth.

Architecture checklist

• Define request and session resource owners.
• Cap transcript bytes held in memory.
• Propagate cancellation to child tasks.
• Set byte limits on caches.
• Run multi-hour cancellation-heavy soak tests.
• Capture profiles before recycling anomalous workers.

Related concepts

• 12 — State Persistence
• 17 — Agent Lifespan Management
• 19 — Context Condensation
• 53 — Health Check Endpoints for Agents
• 71 — Context Window Optimization
• 88 — vLLM / TensorRT-LLM Engines

Primary references

• Kubernetes: Liveness, readiness and startup probes
• Python documentation: tracemalloc