Prefix changes on every request
Detect: Reported cache reuse remains low despite repeated tasks.
Recover: Remove unnecessary volatile fields from the reusable prefix.
Model Optimization, Deployment & Infrastructure
99 — Context Caching (Anthropic/OpenAI Feature)
Reuse provider-supported computation for stable prompt prefixes while preserving fresh generation and provider-specific eligibility, retention, and billing rules.
Concept overview
Context caching, often called prompt or prefix caching, reuses intermediate computation for an eligible repeated prefix. The provider still generates a new response for the request. It differs from semantic caching, which returns a previously generated answer for a similar question, and from durable agent memory, which stores application state.
OpenAI and Anthropic expose provider- and model-specific caching behavior. Eligibility thresholds, prefix matching, controls, retention, routing, and usage accounting can differ and change over time. OpenAI documents model-dependent cache controls and reporting; Anthropic documents explicit and automatic caching options with cache-control semantics. Verify the exact model and deployment documentation instead of assuming one provider’s parameters or lifetime apply to another.
Why it matters in production
Proof of concept
A prototype repeats identical stable prefixes and confirms reuse through reported usage rather than latency alone.
Production
Production uses deterministic prompt serialization, separates stable from changing segments, and records cache read, write, and miss behavior.
Enterprise scale
Enterprise scale evaluates retention and data policies, request locality, template-version churn, and net economics including any cache-write charges.
Architecture components
Stable prefix builder
Serializes reusable instructions, tool definitions, and approved reference material consistently.
Provider cache adapter
Applies the supported model-specific controls without pretending all providers share one API.
Dynamic suffix assembler
Appends changing user data and current task evidence after reusable segments where appropriate.
Usage collector
Records provider-reported cached usage and relevant billing categories.
Cache effectiveness monitor
Tracks reuse, prefix changes, retention behavior, and total accepted-task cost.
The architecture, mapped
flowchart LR
S["Stable instructions and tools"] --> P["Deterministic prefix"]
P --> C["Provider cache eligibility and lookup"]
U["Current user task and evidence"] --> D["Dynamic suffix"]
C --> M["Fresh model generation"]
D --> M
M --> A["Validated new answer"]
M --> R["Cache usage and cost records"]Runtime flow
- 1
Check provider eligibility
Verify supported model, prefix requirements, controls, and applicable data-retention policy.
- 2
Identify reusable content
Separate stable instructions and schemas from timestamps, user-specific facts, and changing retrieval.
- 3
Serialize consistently
Keep segment order and bytes stable where the provider requires matching prefixes.
- 4
Apply supported controls
Use the provider’s documented caching mechanism and version the adapter configuration.
- 5
Submit a fresh request
Send the dynamic suffix and treat the output as a new generation requiring normal validation.
- 6
Measure actual reuse
Inspect reported usage and compare net cost and latency with a no-reuse baseline.
Repeated analysis with a stable schema catalog
Enterprise analytics · Illustrative enterprise scenario, not a reported customer outcome.
Business situation
An analytics assistant uses a large approved schema description across many separate questions.
The problem
The same schema instructions are repeatedly processed, but timestamps inserted near the beginning prevent prefix reuse.
Agent requirement
Reduce repeated prefill work while keeping each query’s data and answer current.
Architecture
A deterministic prefix contains the versioned schema and tool definitions; current task data appears in a separate suffix.
Runtime
- Requests sharing the same approved schema revision become eligible for provider reuse.
- Usage records confirm whether cache reads occurred and whether overall cost improved.
Failure scenario
When the schema changes, the prefix version changes intentionally; correctness takes priority over retaining the previous cache hit rate.
Operational impact
Measure cached-token share, write and read cost, time to first output, and template churn; a warm prefix is an optimization, not an availability guarantee.
Architecture tradeoffs
| Decision | Benefit | Cost |
|---|---|---|
| Stable large prefixes | Can reduce repeated prefill work. | May retain unnecessary context if optimized without task relevance. |
| Longer supported retention | Can improve reuse across wider intervals. | May change cost and data-policy implications. |
| Provider abstraction | Centralizes usage and configuration handling. | Must preserve real capability differences instead of hiding them. |
Failure scenarios & recovery
Assumed cache lifetime is wrong
Detect: Expected hits become misses under the selected model.
Recover: Verify current retention semantics and size capacity for misses.
Common architecture mistakes
Treating caching as semantic answer reuse
Keep normal generation and output validation.
Copying cache parameters across providers
Use explicit provider-specific adapters.
Assuming cache hits from faster latency
Inspect documented usage fields and reconcile cost.
When to choose this pattern
Choose it when
- Use for repeated stable instructions and tool schemas.
- Use for recurring work over the same approved reference prefix.
- Use when provider-reported reuse yields net benefit.
Avoid it when
- Avoid adding irrelevant text merely to increase cached tokens.
- Do not assume cached content is durable memory.
- Avoid retention settings incompatible with the data policy.
Architecture checklist
- Verify the exact provider and model contract.
- Serialize stable prefixes deterministically.
- Separate volatile suffix content.
- Version schema and tool definitions.
- Record cache read and write usage.
- Benchmark misses and reconcile net cost.