72 — Self-RAG / Self-Reflection

Concept overview

Self-RAG refers to a research approach that trains a model to retrieve and critique using reflection signals. In enterprise architectures, the broader pattern is a controlled loop that checks whether evidence supports an answer and retrieves again when a specific gap remains. A generic prompt asking a model to critique itself is not equivalent to the trained Self-RAG method.

Reflection is useful only when it changes an observable decision: seek missing evidence, revise an unsupported claim, abstain, or request review. The same model may repeat its own error, so external evidence, deterministic validation, and held-out evaluation remain necessary. Set iteration and cost limits to prevent endless self-approval or repeated searching.

Why it matters in production

Proof of concept

A prototype compares one-pass answers with a bounded critique loop on labeled unsupported-claim cases.

Production

Production defines critique fields, retrieval triggers, stopping rules, and separate evidence-support checks.

Enterprise scale

At scale, evaluate critique precision and correction success by domain, and reserve extra iterations for requests where they improve accepted outcomes.

Architecture components

Draft generator

Produces an answer with claim-level source references.

Evidence critic

Flags unsupported, conflicting, or missing evidence using a defined rubric.

Retrieval controller

Turns specific evidence gaps into bounded follow-up searches.

Revision stage

Updates claims using newly retrieved evidence without hiding unresolved gaps.

Stopping policy

Caps iterations and selects accept, abstain, or escalate outcomes.

The architecture, mapped

flowchart TD
 E["Initial evidence"] --> D["Draft with claim citations"]
 D --> C["Evidence critique"]
 C --> A["Accept supported answer"]
 C --> G["Specific evidence gap"]
 G --> R["Bounded follow-up retrieval"]
 R --> D
 G --> S["Budget exhausted: abstain or review"]

stateDiagram-v2
 [*] --> Drafting
 Drafting --> Checking
 Checking --> Accepted: Support criteria pass
 Checking --> Retrieving: Specific gap and budget remains
 Retrieving --> Drafting: New evidence found
 Retrieving --> Abstained: No useful evidence
 Checking --> Review: Budget exhausted or sources conflict
 Accepted --> [*]
 Abstained --> [*]
 Review --> [*]

Runtime flow

1. 1

   Draft from available evidence

   Generate a provisional answer and map factual claims to sources.

2. 2

   Evaluate support

   Check whether each important claim follows from accessible evidence.

3. 3

   Identify a concrete gap

   Convert a failed support check into an explicit retrieval objective.

4. 4

   Retrieve selectively

   Search for the missing evidence under unchanged access and temporal constraints.

5. 5

   Revise and compare

   Correct or remove unsupported claims and preserve remaining uncertainty.

6. 6

   Stop deliberately

   Accept only if the rubric passes; otherwise abstain or escalate when the budget ends.

Checking a permit-summary draft

Environmental consulting · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A consulting assistant summarizes conditions from an approved facility permit and amendments.

The problem

An initial draft omits an amendment that changes one reporting condition.

Agent requirement

Detect unsupported or incomplete claims and retrieve the relevant amendment before delivery.

Architecture

A claim-evidence critic checks the draft against versioned permit sources and triggers targeted amendment retrieval.

Runtime

• The critic flags a reporting-frequency claim with incomplete amendment coverage.
• The retriever obtains the applicable amendment and the draft is revised with an explicit citation.

Failure scenario

If contradictory source versions cannot be resolved within the budget, the assistant marks the condition unresolved for expert review.

Operational impact

Measure unsupported-claim reduction, successful correction rate, and extra retrieval cost; a longer deliberation trace is not itself evidence of improved accuracy.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Calling every self-critique loop Self-RAG

Distinguish the trained research method from an application-level pattern.

Using confidence as correctness

Evaluate against evidence and labeled judgments.

Allowing unlimited revisions

Set iteration, token, and elapsed-time limits.

When to choose this pattern

Choose it when

• Use when unsupported claims are a measured failure.
• Use when missing evidence can be retrieved selectively.
• Use for tasks that tolerate bounded additional latency.

Avoid it when

• Avoid for deterministic lookups already validated by schema.
• Do not use self-critique as the only safety control.
• Avoid loops without clear acceptance and stopping criteria.

Architecture checklist

• Define claim-support criteria.
• Map claims to real source passages.
• Specify retrieval triggers.
• Cap loop iterations and cost.
• Detect repeated evidence gaps.
• Evaluate correction success against a one-pass baseline.

Related concepts

• 10 — Reflection Loops
• 18 — Semantic Condition Checking
• 28 — Deterministic Data Grounding
• 31 — LLM Evaluation Frameworks (TruLens/Ragas)
• 46 — Max Iterations Fallback
• 74 — Hypothetical Document Embeddings (HyDE)

Primary references

• Self-RAG: Learning to retrieve, generate and critique