56 — Load Balancing LLM Requests

Concept overview

LLM load balancing assigns requests to compatible inference replicas or provider deployments. Request count is an inadequate load signal: a short classification and a long document analysis consume very different prefill, decoding, and key-value cache resources. A useful balancer estimates work while preserving fairness and admission limits.

The routing unit is normally a complete generation. Moving an active sequence requires engine-specific state transfer and should not be assumed. Prefix-aware routing may improve reuse, but excessive affinity can overload one replica while others remain idle. Balance locality against queueing delay using measured workload behavior.

Why it matters in production

Proof of concept

Start by comparing round-robin routing with observed prompt lengths and completion times.

Production

Production adds bounded queues, token-aware concurrency, cancellation propagation, and separate interactive and background capacity.

Enterprise scale

At scale, route across failure domains, account for heterogeneous accelerators, and test hot-prefix traffic and tenant fairness under sustained overload.

Architecture components

Request estimator

Estimates input tokens, output budget, required model, and deadline.

Replica directory

Publishes available model versions, health, and capacity classes.

Scheduling policy

Combines queue delay, token pressure, fairness, and optional prefix locality.

Admission queue

Bounds waiting work and rejects requests that cannot meet their deadlines.

Inference replicas

Report active tokens, cache pressure, completion, and cancellation.

The architecture, mapped

flowchart LR
 R["Incoming generation"] --> E["Token and deadline estimate"]
 E --> Q["Fair admission queue"]
 Q --> B["Replica selection"]
 B --> A["Replica A short queue"]
 B --> C["Replica B warm prefix"]
 B --> D["Replica C long contexts"]
 A --> M["Load feedback"]
 C --> M
 D --> M
 M --> B

Runtime flow

1. 1

   Classify work

   Resolve model compatibility and separate latency-sensitive traffic from deferred jobs.

2. 2

   Estimate demand

   Count prompt tokens and use an explicit output ceiling to approximate sequence cost.

3. 3

   Select replica

   Choose among healthy compatible replicas using current queue and memory pressure.

4. 4

   Admit or defer

   Reserve capacity or return a clear overload response before an unbounded queue forms.

5. 5

   Run generation

   Keep the sequence on its chosen replica and propagate caller cancellation promptly.

6. 6

   Update estimates

   Compare predicted and actual cost so future routing reflects real request distributions.

Balancing interactive drafting and overnight summaries

Enterprise SaaS · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A document platform shares model capacity between interactive drafting and large workspace summaries.

The problem

Long summary requests occupy cache and delay short user edits.

Agent requirement

Protect interactive response time while using spare capacity for background work.

Architecture

The gateway maintains separate queues and routes with token budgets and per-tenant fairness.

Runtime

• A short edit enters the interactive queue with a small output cap.
• An overnight summary is admitted only when its capacity allocation is available.

Failure scenario

When one replica reports cache exhaustion, new long requests defer while active sequences finish safely.

Operational impact

Evaluate tail latency by workload class, queue age, and accelerator utilization together; average throughput can hide interactive starvation.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Balancing only request counts

Include token and memory pressure in decisions.

Retrying overload immediately

Return bounded retry guidance or defer through a durable queue.

Ignoring cancellation

Release reservations and stop abandoned generations.

When to choose this pattern

Choose it when

• Use for multiple compatible inference replicas.
• Use when prompt and output sizes vary substantially.
• Use when interactive and batch workloads compete.

Avoid it when

• Avoid complex routing for one lightly loaded endpoint.
• Do not move live sequences without supported state transfer.
• Do not prioritize cache hits above authorization or deadlines.

Architecture checklist

• Measure input and output length distributions.
• Set queue and token admission limits.
• Separate workload service objectives.
• Expire replica health information.
• Test fairness during overload.
• Track cancelled work that still consumes compute.

Related concepts

• 14 — Token-Aware Routing
• 33 — Token Rate Limiting
• 52 — Model Redundancy / Multi-Provider Failover
• 59 — Cascade Failure Prevention
• 88 — vLLM / TensorRT-LLM Engines
• 99 — Context Caching (Anthropic/OpenAI Feature)

Primary references

• NVIDIA Triton: TensorRT-LLM backend
• AWS Builders Library: Timeouts, retries and backoff with jitter