47 — Fallback Prompt Strategies

Concept overview

Fallback prompt strategies change how a task is presented when the primary prompt fails in a recognizable way. Examples include a simpler extraction prompt after formatting failure, a narrower evidence question after an overbroad answer, or a clarification request when required input is absent. Fallback is a controlled branch with its own evaluation, not an improvised attempt to bypass a refusal.

Keep mandatory constraints invariant across prompts. Authorization, sensitive-data rules, required citations, and factual uncertainty cannot disappear because a shorter prompt seems more likely to succeed. Classify the failure first: service outages call for operational recovery, while missing evidence calls for retrieval or abstention rather than different wording.

Why it matters in production

Proof of concept

A prototype can compare one alternate prompt against the primary on a known defect set. Include normal cases to detect regressions introduced by the fallback.

Production

Production maintains a versioned prompt registry, explicit trigger conditions, bounded attempts, and shared output validation. Record which strategy actually produced the accepted result.

Enterprise scale

At scale, monitor fallback frequency and total cost. A fallback that handles most traffic may indicate that it should become the primary, while a low-success branch should be removed rather than retained as ritual complexity.

Architecture components

Failure classifier

Identifies formatting, ambiguity, coverage, or other explicitly recoverable prompt-level defects.

Prompt registry

Stores approved primary and fallback templates with versions and invariant constraints.

Strategy selector

Chooses a tested alternative for the observed failure class.

Shared validator

Applies the same required schema, evidence, and policy checks to every strategy.

Outcome recorder

Measures branch usage, recovery success, added cost, and introduced regressions.

The architecture, mapped

flowchart LR
 P["Primary prompt"] --> V["Shared acceptance validator"]
 V --> A["Accepted result"]
 V --> C["Failure classification"]
 C --> F["Approved fallback prompt"]
 C --> H["Clarification or escalation"]
 F --> V
 I["Invariant policy and evidence requirements"] --> P
 I --> F
 B["Strategy attempt budget"] --> C

Runtime flow

1. 1

   Run primary strategy

   Execute the normal prompt with the task's evidence and required constraints.

2. 2

   Inspect failure

   Determine whether the defect can plausibly be repaired through a different prompting approach.

3. 3

   Select approved fallback

   Choose the mapped strategy without weakening policy or inventing missing data.

4. 4

   Reassemble context

   Preserve essential evidence and constraints while simplifying irrelevant instructions.

5. 5

   Validate alternative

   Apply the same acceptance tests used for the primary result.

6. 6

   Stop or promote result

   Accept the supported output or terminate after the bounded strategy budget is exhausted.

Extracting fields from an unusual bill of lading

Logistics documentation · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A freight forwarder's assistant extracts shipment fields from documents with inconsistent layouts and multilingual notes.

The problem

The primary prompt occasionally mixes explanatory prose into a strict structured result or confuses optional fields with required ones.

Agent requirement

The assistant must recover formatting and scope errors without guessing missing shipment identifiers.

Architecture

A failure classifier selects a field-by-field extraction prompt only for known schema or coverage defects, using the same source passages.

Runtime

• The primary output fails a required-field schema check; the fallback asks for each field with a supporting span or explicit unknown.
• The shared validator accepts the corrected structure only when values remain supported by the document.

Failure scenario

If the shipment identifier is absent, the fallback returns unknown and requests clarification instead of manufacturing a plausible number.

Operational impact

Track fallback recovery by defect class, unsupported values, additional tokens, and document-type coverage. Prompt changes are justified by measured recovery rather than stylistic preference.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Using fallback to bypass policy refusals

Preserve policy and distinguish legitimate clarification from evasion.

Trying many prompts without a budget

Limit strategy attempts and reserve finalization capacity.

Treating missing facts as a wording problem

Retrieve evidence or return unknown.

When to choose this pattern

Choose it when

• A known prompt-level defect has a tested alternate strategy.
• All strategies can share clear acceptance criteria.
• Recovery benefit exceeds the added latency and cost.

Avoid it when

• The failure is missing authority or unavailable evidence.
• Alternate prompts weaken mandatory controls.
• No evaluation demonstrates improved accepted outcomes.

Architecture checklist

• Classify fallback triggers explicitly.
• Version primary and alternate templates.
• Preserve invariant policy and evidence rules.
• Use shared acceptance validation.
• Cap total strategy attempts.
• Measure recovery and regression by defect class.

Related concepts

• 20 — Deterministic Output Stripping
• 28 — Deterministic Data Grounding
• 43 — Self-Correction Loops
• 45 — Graceful Degradation
• 46 — Max Iterations Fallback

Primary references

• LangGraph: workflows and agents
• JSON Schema: understanding JSON Schema