95 — Distributed Memory Architectures

Concept overview

Distributed memory architectures in model infrastructure address the memory required to execute a model across multiple accelerators or memory tiers. Tensor parallelism partitions tensor computation, pipeline parallelism partitions layers, and other schemes distribute experts or separate prefill from decoding. This topic concerns model execution memory, distinct from an agent’s durable conversational memory.

Memory does not become a free shared pool. Partitioning introduces communication, synchronization, topology constraints, and coordinated failure. CPU offload or remote cache transfer trades scarce accelerator memory for bandwidth and latency. Choose a strategy after estimating weights, activations, key-value cache, and runtime overhead for the target context and concurrency.

Why it matters in production

Proof of concept

A prototype inventories memory components and tests whether quantization or smaller batches can meet the requirement on one device.

Production

Production selects a supported partitioning strategy, benchmarks communication, and treats distributed workers as a coordinated serving group.

Enterprise scale

Enterprise scale plans topology-aware placement, failure domains, cache movement, and capacity accounting across heterogeneous nodes and workloads.

Architecture components

Model partition planner

Maps layers, tensors, or experts to supported device groups.

Accelerator workers

Hold their assigned weights and execution state.

Communication fabric

Carries activations, reductions, or cache transfers between workers.

Distributed scheduler

Coordinates request execution and group-level admission.

Memory telemetry

Separates weights, cache, activations, offloaded state, and communication overhead.

The architecture, mapped

flowchart LR
 Q["Inference request"] --> S["Group scheduler"]
 S --> A["Device A model partition"]
 A --> B["Device B model partition"]
 B --> C["Device C model partition"]
 C --> O["Output tokens"]
 A --> M["Memory and fabric telemetry"]
 B --> M
 C --> M
 M --> S

sequenceDiagram
 participant S as Group scheduler
 participant A as Device A
 participant B as Device B
 participant F as Communication fabric
 S->>A: Execute token step
 S->>B: Execute token step
 A->>F: Partial tensor result
 B->>F: Partial tensor result
 F-->>A: Coordinated reduction result
 F-->>B: Coordinated reduction result
 A-->>S: Step complete
 B-->>S: Step complete

Runtime flow

1. 1

   Build the memory model

   Estimate required capacity for the actual precision, context length, and active sequences.

2. 2

   Choose partitioning

   Select tensor, pipeline, expert, or supported disaggregated execution based on the bottleneck.

3. 3

   Place the worker group

   Use compatible hardware and a network topology adequate for required communication.

4. 4

   Load and synchronize

   Verify partition identities and initialize communication before publishing readiness.

5. 5

   Execute coordinated inference

   Move required activations or reductions while maintaining consistent sequence state.

6. 6

   Recover the group

   On worker failure, stop affected traffic and restart or reconstruct state according to engine support.

Serving a large technical-analysis model

Scientific publishing · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A research platform needs a model and context configuration that exceed one accelerator’s usable memory.

The problem

Naive CPU offload fits the weights but makes interactive analysis too slow.

Agent requirement

Select a supported multi-device layout that meets both capacity and latency requirements.

Architecture

The team profiles weights and cache, then benchmarks partitioned serving on a high-bandwidth worker group.

Runtime

• A request is admitted only when the whole serving group is ready.
• Telemetry separates communication time from model compute and tracks cache growth with context length.

Failure scenario

If one worker fails, the group stops accepting requests and incomplete generations restart under an explicit client policy.

Operational impact

Measure per-request latency, accepted throughput, memory headroom, and communication fraction; adding devices can increase cost without improving latency.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Confusing model memory with agent memory

Separate inference partitioning from durable workflow state.

Sizing only model weights

Include KV cache, activations, and runtime overhead.

Assuming more devices always improve speed

Benchmark topology and communication under real concurrency.

When to choose this pattern

Choose it when

• Use when a validated model configuration exceeds one device.
• Use when supported parallelism improves measured capacity or throughput.
• Use with adequate communication infrastructure and operational expertise.

Avoid it when

• Avoid distribution when one device meets the objective more simply.
• Do not assume arbitrary devices form efficient shared memory.
• Avoid unsupported live-state migration promises.

Architecture checklist

• Account for every memory component.
• Choose an engine-supported partition strategy.
• Benchmark interconnect overhead.
• Place complete compatible worker groups.
• Test coordinated worker failure.
• Compare cost and latency against simpler alternatives.

Related concepts

• 57 — Crash-Safe Memory
• 84 — Model Quantization (AWQ/GGUF)
• 88 — vLLM / TensorRT-LLM Engines
• 90 — LoRA / QLoRA Fine-Tuning
• 91 — GPU Cluster Orchestration (Kubernetes/KServe)
• 99 — Context Caching (Anthropic/OpenAI Feature)

Primary references

• NVIDIA Triton: TensorRT-LLM backend
• vLLM parallelism and scaling