31 — LLM Evaluation Frameworks (TruLens/Ragas)

Concept overview

Evaluation frameworks organize datasets, execution traces, scoring, and comparisons across application versions. TruLens and Ragas provide useful evaluation capabilities, but the architecture must still define what success means for the product. A metric is a measurement instrument with assumptions, not an automatic release decision.

For RAG, distinguish retrieval relevance, answer relevance, and support by retrieved evidence. Faithfulness to a retrieved passage does not prove that the passage is true or current. For agents, add tool selection, argument correctness, authorization, side-effect outcomes, and completion criteria. Evaluate the trajectory when the final answer hides an unsafe or wasteful path.

Why it matters in production

Proof of concept

A prototype needs a small representative set with reviewed expected behavior. Include unanswerable questions and tool failures, not only successful demonstrations.

Production

Production versions datasets, prompts, models, judge configurations, and scoring code. Calibrate model judges against human labels and retain uncertainty rather than treating every score as exact.

Enterprise scale

At scale, combine offline release tests with sampled production evaluation. Segment results by tenant, language, intent, and risk so aggregate improvements cannot hide critical regressions.

Architecture components

Evaluation dataset

Contains representative inputs, expected behavior, evidence, and meaningful failure cases.

Execution recorder

Captures retrieval, tool calls, model versions, and outcomes with appropriate data minimization.

Metric pipeline

Computes deterministic checks and model-assisted scores for distinct quality dimensions.

Human calibration set

Establishes agreement and failure modes of automated judges.

Release gate

Applies risk-weighted thresholds and compares candidates with a versioned baseline.

The architecture, mapped

flowchart LR
 D["Versioned case dataset"] --> R["Candidate runner"]
 R --> T["Execution traces"]
 T --> M["Component and outcome metrics"]
 H["Human calibration labels"] --> M
 M --> C["Baseline comparison"]
 C --> G["Release criteria"]
 G --> P["Production sampling"]
 P --> D

Runtime flow

1. 1

   Define quality

   Translate product requirements into observable success and failure criteria.

2. 2

   Build cases

   Sample realistic workloads and add adversarial, missing-evidence, and degraded-dependency examples.

3. 3

   Run candidates

   Execute identical cases with pinned application and data versions where feasible.

4. 4

   Score dimensions

   Separate retrieval, grounding, task completion, safety, latency, and cost measurements.

5. 5

   Calibrate judgments

   Review disagreements and estimate uncertainty for model-based metrics.

6. 6

   Decide and monitor

   Apply release criteria, preserve evidence, and compare production drift against the baseline.

Evaluating a technical support assistant

Enterprise SaaS · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A software vendor updates retrieval and prompts for an assistant that diagnoses integration errors.

The problem

A higher average answer score can conceal incorrect tool arguments and regressions on rare authentication failures.

Agent requirement

The assistant must resolve supported cases with correct evidence and safe tool usage under practical latency and cost limits.

Architecture

A versioned evaluation suite combines RAG metrics, deterministic tool-contract checks, and analyst-labeled resolution outcomes.

Runtime

• The candidate runs against the same support cases and source snapshot as the current release.
• Reviewers inspect judge disagreements and high-impact regressions before a release decision is recorded.

Failure scenario

If the judge model changes, the team recalibrates scores rather than interpreting a shifted scale as product improvement.

Operational impact

Track resolution quality, unsupported claims, tool accuracy, judge agreement, and cost per successful case. Framework scores are diagnostic signals, not universal quality guarantees.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Using faithfulness as factual truth

Check source authority and task correctness separately.

Changing judge and application simultaneously

Pin or recalibrate the judge to preserve comparability.

Scoring only the final answer

Evaluate tools, evidence, permissions, and external effects.

When to choose this pattern

Choose it when

• Model or retrieval changes need evidence-based comparison.
• Agent behavior has measurable task outcomes.
• Release decisions require repeatable quality and risk signals.

Avoid it when

• A framework is being used without product-specific criteria.
• No representative data or review capacity exists.
• One aggregate score is expected to prove system safety.

Architecture checklist

• Version datasets and scoring configuration.
• Include unanswerable and failure cases.
• Calibrate judges against reviewed labels.
• Segment critical intents and populations.
• Measure trajectory and final outcome.
• Record release thresholds and uncertainty handling.

Related concepts

• 10 — Reflection Loops
• 28 — Deterministic Data Grounding
• 32 — Model Bias Auditing
• 35 — Adversarial Prompt Testing
• 93 — Model Versioning and A/B Testing

Primary references

• Ragas: faithfulness evaluation
• TruLens: the RAG triad