85 — LLM Cold Start Optimization

Concept overview

LLM cold start is the delay before a new serving instance can accept useful work. It includes scheduling, image retrieval, model download, deserialization, accelerator allocation, kernel compilation or graph capture, tokenizer setup, and warmup. Optimizing only container startup can leave model readiness unchanged.

The right strategy depends on demand and infrastructure: cached artifacts, smaller images, prebuilt compatible engines, local model storage, and a minimum warm pool address different stages. Readiness must reflect actual inference capability, while startup probes allow legitimate initialization without restart loops. A warm process with an unloaded model is not warm capacity.

Why it matters in production

Proof of concept

A prototype records timestamps for every initialization stage and runs one representative warmup request.

Production

Production pins artifacts, caches model files safely, separates startup from readiness, and keeps enough warm capacity for the latency objective.

Enterprise scale

Enterprise scale coordinates model placement, rollout surge capacity, cache invalidation, and regional warm pools without exhausting accelerator quota.

Architecture components

Artifact cache

Provides verified images and model weights near the serving node.

Placement scheduler

Selects compatible accelerators and available memory.

Runtime initializer

Loads weights and prepares kernels and execution state.

Warmup runner

Exercises representative shapes and verifies a usable output.

Readiness controller

Admits traffic only after initialization and capacity checks pass.

The architecture, mapped

flowchart LR
 S["Schedule accelerator"] --> I["Fetch image"]
 I --> W["Load verified model weights"]
 W --> K["Prepare kernels and caches"]
 K --> U["Representative warmup"]
 U --> R["Ready for traffic"]
 C["Node-local artifact cache"] --> I
 C --> W

Runtime flow

1. 1

   Measure startup stages

   Separate scheduling delay, artifact transfer, model loading, and runtime preparation.

2. 2

   Place compatible capacity

   Schedule onto nodes with the required hardware and sufficient headroom.

3. 3

   Load verified artifacts

   Use checksummed local caches or efficient transfer without mixing model revisions.

4. 4

   Prepare the runtime

   Allocate model and cache memory and compile supported execution paths as needed.

5. 5

   Warm representative requests

   Exercise realistic prompt shapes and verify output and resource stability.

6. 6

   Publish readiness

   Add the replica to routing only when it can meet its initial admission contract.

Preparing capacity for scheduled campaigns

Retail operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A retailer expects bursts of assistant traffic when new promotions go live.

The problem

Reactive scaling starts model replicas too late, causing queues during the first minutes of demand.

Agent requirement

Have usable inference capacity ready before the predictable burst without keeping the entire peak fleet warm all day.

Architecture

A scheduled capacity plan preloads approved artifacts and warms a bounded replica pool.

Runtime

• Before campaign launch, nodes cache the model and run representative requests.
• The router admits traffic only after readiness and token-capacity checks pass.

Failure scenario

If a new artifact fails warmup, existing warm replicas continue serving while the rollout pauses.

Operational impact

Measure time to usable capacity, first-request latency, and idle warm-pool cost; container-running status is not the service objective.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Measuring only process start

Instrument the full path to successful inference.

Sending traffic before warmup

Keep readiness false until representative execution passes.

Scaling from utilization alone

Include queue growth and startup lead time.

When to choose this pattern

Choose it when

• Use for autoscaled model endpoints.
• Use for predictable demand bursts.
• Use when model loading dominates availability of new capacity.

Avoid it when

• Avoid expensive warm pools for delay-tolerant sparse jobs.
• Do not reuse compiled artifacts across unsupported hardware.
• Avoid masking failed warmup with permissive readiness.

Architecture checklist

• Time every initialization stage.
• Pin and verify cached artifacts.
• Warm representative request shapes.
• Configure separate startup and readiness checks.
• Budget rollout surge accelerator capacity.
• Measure idle cost against latency benefit.

Related concepts

• 53 — Health Check Endpoints for Agents
• 82 — Small Language Models (SLMs) Deployment
• 88 — vLLM / TensorRT-LLM Engines
• 89 — Serverless LLM Architecture
• 91 — GPU Cluster Orchestration (Kubernetes/KServe)
• 99 — Context Caching (Anthropic/OpenAI Feature)

Primary references

• Kubernetes: Liveness, readiness and startup probes
• NVIDIA Triton: TensorRT-LLM backend