61 — Semantic Caching (Redis/GPTCache)

Concept overview

Semantic caching retrieves a previous answer by meaning rather than exact prompt bytes. It typically embeds the incoming query, searches cached query vectors, and applies an acceptance threshold before returning a stored response. The important contract is answer equivalence: two similar questions are not necessarily entitled to the same answer.

A cache entry must carry more than a vector and text. Tenant scope, authorization context, locale, model or prompt revision, source versions, and expiration determine eligibility. Personalized balances, changing prices, and action requests are usually poor candidates. Semantic caching differs from provider prefix caching, which reuses computation while still generating a fresh answer.

Why it matters in production

Proof of concept

A prototype measures hit quality on repeated public questions and labels false matches manually.

Production

Production adds metadata gates, per-intent thresholds, invalidation, provenance, and a bypass for sensitive or volatile tasks.

Enterprise scale

At enterprise scale, track false-hit rates by tenant and language, control cache cardinality, and test revocation propagation independently of expiration.

Architecture components

Eligibility classifier

Excludes personalized, action-oriented, or rapidly changing requests.

Embedding service

Maps eligible questions into a versioned similarity space.

Scoped vector cache

Stores answer evidence and filters entries by tenant and policy metadata.

Acceptance gate

Checks semantic threshold, freshness, and answer compatibility.

Generation path

Produces and validates fresh answers on a miss before storing eligible results.

The architecture, mapped

flowchart LR
 Q["Question"] --> E["Eligibility gate"]
 E --> V["Scoped vector lookup"]
 V --> A["Similarity and freshness checks"]
 A --> H["Reuse validated answer"]
 A --> G["Generate fresh answer"]
 E --> G
 G --> S["Store eligible evidence-backed response"]
 S --> V

Runtime flow

1. 1

   Check eligibility

   Determine whether answer reuse is allowed for the request type and data classification.

2. 2

   Build scope

   Resolve tenant, language, policy revision, and source-version constraints.

3. 3

   Search candidates

   Embed the question and retrieve nearby entries only within the allowed scope.

4. 4

   Validate reuse

   Apply calibrated similarity and freshness checks; treat uncertainty as a miss.

5. 5

   Generate on miss

   Run retrieval and generation, validate the answer, and retain its supporting evidence.

6. 6

   Store and observe

   Write a bounded-life entry and measure hits, misses, false matches, and invalidation lag.

Repeated passenger policy questions

Public transport · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A transit operator receives many paraphrases of the same baggage and bicycle questions.

The problem

Repeated model calls answer stable policy questions while personalized journey requests share the interface.

Agent requirement

Reuse policy answers without serving one passenger another passenger’s itinerary.

Architecture

The cache permits public policy intents and keys eligibility by policy revision and service region.

Runtime

• A bicycle-policy question matches a validated answer for the same region.
• A fare-balance question bypasses caching and reads the passenger account directly.

Failure scenario

When the policy changes, its revision invalidates old entries even if their similarity scores remain high.

Operational impact

Track avoided generation work alongside false-hit rate and policy freshness; a higher hit rate is harmful if answer equivalence deteriorates.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Treating cosine similarity as correctness

Calibrate thresholds using labeled answer-equivalence pairs.

Caching before output validation

Store only accepted responses with provenance.

Sharing entries across users by default

Partition or gate reuse using explicit authorization scope.

When to choose this pattern

Choose it when

• Use for repeated stable informational questions.
• Use where false hits can be evaluated with labeled examples.
• Use when generation cost exceeds cache lookup overhead.

Avoid it when

• Avoid personalized balances and action requests.
• Avoid rapidly changing facts without reliable invalidation.
• Do not reuse answers across incompatible permissions.

Architecture checklist

• Define eligible request classes.
• Version embeddings and source revisions.
• Calibrate thresholds by intent.
• Test negation and near-match failures.
• Invalidate on permission changes.
• Measure false hits as a first-class metric.

Related concepts

• 21 — Authorization-Aware RAG
• 34 — Tenant Isolation
• 63 — Metadata Pre-Filtering
• 68 — Time-Weighted Vector Retrieval
• 96 — Semantic Similarity Analytics
• 99 — Context Caching (Anthropic/OpenAI Feature)

Primary references

• Redis: Semantic cache architecture