14 — Token-Aware Routing

Concept overview

Token-aware routing uses input size, expected output length, context-window limits, and remaining budget to choose an execution path. The destination may be a smaller model, a larger-context model, a retrieval step, a condensation stage, or an asynchronous job. Token count is a capacity signal, not a reliable measure of task difficulty.

Estimate the complete request: system instructions, tool schemas, retrieved evidence, conversation state, and reserved output all consume capacity. A short request can require difficult reasoning, while a long document may need simple extraction. Apply capability and policy constraints before optimizing token cost.

Why it matters in production

Proof of concept

A prototype can compare token estimates with actual usage across a representative request set. Include multilingual text, tables, and tool-heavy prompts.

Production

Production reserves output and tool-loop capacity, rejects impossible requests early, and reconciles estimated versus billed usage. Use model-specific tokenization where available rather than a universal characters-per-token rule.

Enterprise scale

At scale, tenant budgets and provider quotas shape routing decisions. Track truncation, fallback frequency, and downstream correction cost so apparently cheap routes do not increase total cost per successful task.

Architecture components

Prompt estimator

Counts or estimates all input segments and reserves expected output capacity.

Capability filter

Removes models and paths that cannot meet task, policy, or residency requirements.

Budget allocator

Assigns per-step token ceilings from the remaining task allowance.

Routing policy

Selects extraction, retrieval, condensation, or model execution using validated thresholds.

Usage reconciler

Records actual consumption and updates estimates without changing completed accounting.

The architecture, mapped

flowchart LR
 Q["Request and evidence"] --> E["Full prompt estimator"]
 E --> C["Capability and policy filter"]
 C --> S["Budget strategy"]
 S --> L["Compact extraction path"]
 S --> R["Retrieve relevant passages"]
 S --> M["Larger context model"]
 L --> U["Usage and quality ledger"]
 R --> U
 M --> U
 B["Remaining task budget"] --> S

Runtime flow

1. 1

   Measure request

   Estimate instructions, schemas, history, evidence, and expected output using the candidate model's tokenizer.

2. 2

   Check feasibility

   Verify context fit and reserve capacity for mandatory validation or additional tool observations.

3. 3

   Filter capability

   Exclude paths that lack required language, reasoning, structure, or data-handling support.

4. 4

   Select strategy

   Choose direct execution, smaller context retrieval, condensation, or a suitable model.

5. 5

   Enforce limits

   Apply output ceilings and stop optional iterations before exhausting the parent budget.

6. 6

   Reconcile outcome

   Record actual tokens, quality, and fallback cost to evaluate routing economics.

Processing broker submission packages

Insurance · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A specialty insurer receives submissions ranging from short forms to lengthy engineering reports with scanned appendices.

The problem

Routing by file size alone sends difficult short submissions to weak models and wastes large-context calls on irrelevant attachments.

Agent requirement

The assistant must extract specified underwriting facts with source references under a per-submission processing budget.

Architecture

A tokenizer-aware estimator combines document structure with capability requirements and routes relevant sections to extraction or deeper review.

Runtime

• The ingestion service identifies attachments and estimates prompt demand; retrieval selects evidence for each required field.
• Simple tables use a compact extraction path, while ambiguous exclusions use a stronger eligible model with reserved verification capacity.

Failure scenario

If estimates understate a scanned report's token demand, the system partitions evidence by semantic section rather than silently truncating required material.

Operational impact

Measure tokens per accepted field, context overflow, unsupported extraction, and total fallback cost. A lower model bill is useful only if review quality remains acceptable.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Equating short prompts with easy tasks

Evaluate task capability independently from length.

Truncating evidence to force context fit

Retrieve or partition while preserving required coverage.

Ignoring reserved output tokens

Budget input, output, and follow-up observations together.

When to choose this pattern

Choose it when

• Requests vary substantially in context and output size.
• Several capable execution paths have different token economics.
• Task-level spend and context limits must be enforced.

Avoid it when

• One model and fixed-size prompts already meet constraints.
• Quality cannot be measured across candidate routes.
• Routing overhead exceeds plausible token savings.

Architecture checklist

• Count system and tool-schema tokens.
• Reserve output and validation capacity.
• Use candidate-specific context limits.
• Separate capability from prompt length.
• Reconcile estimates with actual usage.
• Evaluate total cost after escalations.

Related concepts

• 5 — Dynamic Routing
• 19 — Context Condensation
• 71 — Context Window Optimization
• 81 — Dynamic Model Routing
• 94 — Token Cost Allocation Tracking

Primary references

• LangGraph: workflows and agents