91 — GPU Cluster Orchestration (Kubernetes/KServe)

Concept overview

GPU cluster orchestration schedules model workloads onto compatible accelerators and maintains usable serving capacity through failures and demand changes. Kubernetes provides resource scheduling and lifecycle primitives; serving layers such as KServe add model-oriented deployment abstractions. Actual capabilities depend on the selected installation, runtime, and scaling integration.

Accelerator topology, memory, driver compatibility, artifact loading, and distributed worker coordination shape placement. A model requiring several devices may need all workers available together, while independent replicas can scale separately. Autoscaling must consider queueing and active token demand because utilization alone can remain high or low for misleading reasons.

Why it matters in production

Proof of concept

A prototype validates one model on the intended node type and confirms device allocation and readiness behavior.

Production

Production pins drivers and runtime images, separates startup from readiness, and configures disruption and rollout limits.

Enterprise scale

Enterprise scale adds topology-aware placement, quotas, priority classes, regional capacity plans, and recovery procedures for multi-device worker groups.

Architecture components

Cluster scheduler

Assigns pods to compatible nodes using declared accelerator resources and placement constraints.

Device integration

Exposes accelerators and relevant allocation capabilities to the platform.

Model serving controller

Manages model revisions, runtime configuration, and traffic readiness.

Autoscaling signals

Translate backlog, latency, and active work into desired capacity.

Artifact and telemetry services

Supply verified model bundles and observe accelerator, engine, and pod health.

The architecture, mapped

flowchart TD
 D["Serving revision"] --> C["Model serving controller"]
 C --> K["Cluster scheduler"]
 K --> A["Compatible accelerator nodes"]
 A --> W["Model load and warmup"]
 W --> R["Ready inference replicas"]
 R --> Q["Queue and token telemetry"]
 Q --> S["Autoscaling policy"]
 S --> C

Runtime flow

1. 1

   Declare the serving contract

   Specify model bundle, hardware class, memory needs, and replica or distributed-worker structure.

2. 2

   Schedule compatible resources

   Apply node and topology constraints and wait for all required capacity.

3. 3

   Initialize the model

   Load verified artifacts and allow startup time before liveness enforcement.

4. 4

   Publish ready capacity

   Route requests only after warmup and resource admission checks pass.

5. 5

   Scale from workload signals

   Use queue and service metrics with stabilization to account for slow model startup.

6. 6

   Drain and recover

   Stop new work, checkpoint eligible tasks, and replace failed replicas or coordinated worker groups.

Operating a shared private inference fleet

Financial research · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

Several research teams share private model-serving capacity with different context lengths and data policies.

The problem

Large distributed jobs fragment accelerator capacity and interfere with interactive services.

Agent requirement

Place compatible workloads predictably while protecting interactive service objectives.

Architecture

The cluster separates workload priorities, enforces quotas, and schedules coordinated model workers with topology constraints.

Runtime

• An interactive replica receives reserved capacity and completes warmup before entering service.
• A larger research job waits until its full worker group can be placed rather than consuming unusable partial capacity.

Failure scenario

If one distributed worker fails, the serving group is removed from routing and recovered as a coordinated unit.

Operational impact

Measure time to usable capacity, pending-resource age, disruption impact, and accepted throughput; allocated GPUs are not equivalent to healthy serving capacity.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Scaling on CPU alone

Use inference workload and queue metrics.

Marking pods ready before model warmup

Gate traffic on actual serving capability.

Ignoring disruption of active generations

Drain and bound termination behavior explicitly.

When to choose this pattern

Choose it when

• Use for shared or elastic self-hosted accelerator fleets.
• Use for coordinated multi-device model serving.
• Use when repeatable rollout and recovery matter.

Avoid it when

• Avoid cluster complexity for a single stable workstation.
• Do not assume every node has compatible accelerator features.
• Avoid unbounded workloads without tenant quotas.

Architecture checklist

• Pin driver and runtime compatibility.
• Declare accelerator and topology requirements.
• Separate startup and readiness probes.
• Set quotas and priority policies.
• Scale using queue and token signals.
• Test worker-group failure and graceful drain.

Related concepts

• 53 — Health Check Endpoints for Agents
• 56 — Load Balancing LLM Requests
• 85 — LLM Cold Start Optimization
• 88 — vLLM / TensorRT-LLM Engines
• 89 — Serverless LLM Architecture
• 95 — Distributed Memory Architectures

Primary references

• KServe: Generative inference autoscaling examples
• Kubernetes: Liveness, readiness and startup probes
• NVIDIA Triton: TensorRT-LLM backend