81 — Dynamic Model Routing

Concept overview

Dynamic model routing chooses among models according to the request rather than sending all work to one default. A router may consider task type, input modality, context length, required tools, latency deadline, and evaluated difficulty. Cost and latency are optimization objectives only after the model satisfies the task and data-policy contract.

Routing introduces its own error surface. A cheap classifier can underestimate a difficult request, while a complex router can cost more than it saves. Model self-confidence is not a reliable universal complexity score. Start with transparent rules, collect outcome data, and add learned routing only when the evaluation supports it.

Why it matters in production

Proof of concept

A prototype compares a fixed strong baseline with a simple two-route policy on labeled tasks.

Production

Production versions routing decisions, evaluates each route, and defines escalation when output validation fails.

Enterprise scale

Enterprise scale handles capacity, regional eligibility, tenant budgets, and drift while keeping routing changes independently reversible from model changes.

Architecture components

Request profiler

Extracts task requirements and bounded complexity signals.

Capability registry

Records approved modalities, schemas, regions, and evaluated quality.

Routing policy

Chooses the lowest-cost eligible route meeting the service objective.

Outcome validator

Checks task-specific acceptance and triggers permitted escalation.

Decision ledger

Stores route version, selected model, reasons, and actual usage.

The architecture, mapped

flowchart TD
 R["Task requirements"] --> P["Policy and capability filter"]
 P --> C["Complexity and capacity decision"]
 C --> S["Small specialized model"]
 C --> L["Larger reasoning model"]
 S --> V["Task acceptance check"]
 L --> V
 V --> E["Eligible escalation"]
 E --> L
 V --> A["Accepted result"]

Runtime flow

1. 1

   Resolve hard constraints

   Determine data location, required features, and maximum context before optimizing cost.

2. 2

   Profile task difficulty

   Use observable request features and evaluated heuristics rather than ungrounded confidence.

3. 3

   Select eligible route

   Combine capability, current capacity, and the task’s quality floor.

4. 4

   Execute and validate

   Run inference and check schema, evidence, or business acceptance rules.

5. 5

   Escalate selectively

   Use a stronger approved route only when the remaining deadline and budget permit.

6. 6

   Learn from outcomes

   Compare route-level quality and total cost, including classifier and escalation overhead.

Routing routine and unusual correspondence

Insurance operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A service assistant classifies routine claim correspondence and drafts explanations for unusual disputes.

The problem

Using one large model for every label is costly, while a small model struggles with complex exceptions.

Agent requirement

Route routine classification efficiently and reserve stronger reasoning for exception-heavy tasks.

Architecture

A task profiler selects evaluated model routes and a validator checks classification schema and evidence requirements.

Runtime

• A standard document label uses the specialized route.
• An ambiguous dispute with several policy references uses the stronger route or escalates after validation failure.

Failure scenario

If the router version begins underclassifying difficult cases, traffic returns to the fixed baseline while the routing policy is reviewed.

Operational impact

Measure accepted-task cost, escalation rate, and quality by route; nominal per-token savings can disappear after repeated failures.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Optimizing token price alone

Compare total cost per accepted task.

Routing from model confidence only

Use calibrated outcomes and observable features.

Ignoring data-policy compatibility

Apply hard eligibility constraints before selection.

When to choose this pattern

Choose it when

• Use for mixed workloads with distinct capability needs.
• Use when route quality can be evaluated by task class.
• Use when cost savings exceed routing and escalation overhead.

Avoid it when

• Avoid complex routing for uniform low-volume work.
• Do not route sensitive data to unapproved models.
• Avoid unvalidated difficulty classifiers for consequential tasks.

Architecture checklist

• Define route-specific quality floors.
• Version the routing policy.
• Measure total accepted-task cost.
• Include capacity and deadline constraints.
• Test escalation and fallback paths.
• Keep a fixed-baseline rollback route.

Related concepts

• 5 — Dynamic Routing
• 14 — Token-Aware Routing
• 31 — LLM Evaluation Frameworks (TruLens/Ragas)
• 52 — Model Redundancy / Multi-Provider Failover
• 82 — Small Language Models (SLMs) Deployment
• 94 — Token Cost Allocation Tracking

Primary references

• NVIDIA Triton: TensorRT-LLM backend
• AWS Builders Library: Timeouts, retries and backoff with jitter