71 — Context Window Optimization

Concept overview

Context window optimization selects, compresses, and orders information within a model’s input and output limits. A large supported window does not imply that every included token contributes equally to answer quality. Long histories can add irrelevant detail, conflicting instructions, and substantial prefill cost.

The useful abstraction is a context budget with protected and optional segments. System constraints, current task facts, and indispensable evidence receive explicit allocation; old dialogue and redundant retrieval receive selective retention or summarization. Summaries are derived artifacts and should retain provenance, unresolved decisions, and links to recover original detail.

Why it matters in production

Proof of concept

A prototype measures token use and answer quality as conversation length grows.

Production

Production reserves output capacity, compacts history at defined thresholds, and tests whether critical facts survive compaction.

Enterprise scale

Enterprise scale tracks budget use by task, supports versioned summaries, and evaluates long-context alternatives against retrieval and selective reconstruction.

Architecture components

Token estimator

Counts the actual request format using the selected model’s tokenizer or supported estimate.

Segment allocator

Reserves mandatory policy, current task, evidence, and output capacity.

History compactor

Produces bounded summaries with source references and unresolved commitments.

Evidence selector

Removes duplication and chooses passages supporting the requested answer.

Context evaluator

Tests fact retention, contradiction handling, and response completeness.

The architecture, mapped

flowchart TD
 H["Long conversation history"] --> S["Versioned summary"]
 E["Retrieved passages"] --> D["Deduplicate and select"]
 P["Mandatory policy and current task"] --> B["Context allocator"]
 S --> B
 D --> B
 B --> O["Reserved output capacity"]
 B --> V["Token and fact-retention check"]
 V --> M["Model request"]

Runtime flow

1. 1

   Inventory context

   Classify candidate segments by trust, freshness, relevance, and required retention.

2. 2

   Reserve fixed capacity

   Allocate mandatory instructions and an adequate output budget before optional text.

3. 3

   Select evidence

   Keep passages covering distinct answer requirements rather than merely the highest similarity scores.

4. 4

   Compact history

   Summarize older dialogue while preserving decisions, identifiers, and unresolved obligations.

5. 5

   Validate the final request

   Count serialized tokens and confirm protected facts and constraints remain present.

6. 6

   Recover detail when needed

   Retrieve original records if a later task requires information omitted by compaction.

Maintaining a long design-review conversation

Engineering services · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A design assistant follows weeks of decisions about an industrial control upgrade.

The problem

Old alternatives and repeated documents crowd out current constraints and requested analysis.

Agent requirement

Preserve approved decisions and unresolved questions without resending every turn.

Architecture

A decision ledger and versioned conversation summary feed a context allocator with targeted source retrieval.

Runtime

• The current task asks about a previously deferred interface decision.
• The allocator retrieves the original decision record and adds only the relevant discussion around it.

Failure scenario

If the summary conflicts with the authoritative decision ledger, the ledger wins and the summary is rebuilt.

Operational impact

Measure retained-constraint accuracy, input tokens, and complete-answer rate; shrinking the prompt is useful only if required reasoning evidence survives.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Truncating from the oldest token blindly

Preserve commitments and durable identifiers explicitly.

Treating summary text as ground truth

Retain source links and recover originals.

Measuring only prompt size

Evaluate task quality and constraint retention.

When to choose this pattern

Choose it when

• Use for long conversations and large retrieved contexts.
• Use when prefill cost dominates response time.
• Use when repeated evidence crowds out current task facts.

Avoid it when

• Avoid compression when exact wording is essential and fits comfortably.
• Do not remove safety or authorization constraints to save tokens.
• Avoid summaries without recoverable provenance.

Architecture checklist

• Reserve output capacity first.
• Define protected facts and segments.
• Version summaries and their source ranges.
• Deduplicate retrieved passages.
• Test long-horizon fact retention.
• Support retrieval of omitted original details.

Related concepts

• 12 — State Persistence
• 19 — Context Condensation
• 64 — Parent-Child Chunking
• 70 — Dynamic Prompt Assembly
• 78 — Lost in the Middle Mitigation
• 80 — Token Pruning

Primary references

• Lost in the Middle: How Language Models Use Long Contexts
• LLMLingua: Compressing Prompts for Accelerated Inference