93 — Model Versioning and A/B Testing

Concept overview

Model versioning identifies the full behavior-producing bundle: model weights or provider revision, tokenizer, prompt, tools, retrieval configuration, adapter, and runtime settings. A model name alone is insufficient when several surrounding components change. Reproducible evaluation requires knowing exactly which bundle produced each outcome.

A/B testing assigns comparable traffic to alternative bundles to estimate differences in task outcomes. Stable user or workflow assignment avoids mixing behaviors within one experience. Shadow evaluation sends copies for observation without allowing the shadow path to perform external effects. Canary rollout primarily limits operational exposure; it is not automatically a statistically valid experiment.

Why it matters in production

Proof of concept

A prototype compares candidate and baseline on the same held-out task set with deterministic scoring where possible.

Production

Production uses immutable bundle IDs, stable assignment, predefined metrics, and an immediate rollback route.

Enterprise scale

Enterprise scale monitors subgroup regressions, experiment interactions, delayed outcomes, and data-policy eligibility before promoting a candidate.

Architecture components

Behavior registry

Stores immutable model and surrounding configuration lineage.

Assignment service

Routes a stable eligible cohort to baseline or candidate.

Outcome collector

Associates task quality, latency, cost, and failure evidence with the assigned bundle.

Evaluation gate

Applies predefined acceptance criteria and checks important slices.

Release controller

Promotes or rolls back traffic without losing in-flight workflow identity.

The architecture, mapped

flowchart TD
 R["Eligible workflow"] --> A["Stable assignment"]
 A --> B["Baseline behavior bundle"]
 A --> C["Candidate behavior bundle"]
 B --> O["Task outcome collection"]
 C --> O
 O --> E["Quality and operational gates"]
 E --> P["Promote candidate"]
 E --> V["Revert traffic"]

Runtime flow

1. 1

   Register the complete bundle

   Pin the model, prompts, tools, retrieval, and runtime configuration.

2. 2

   Define the comparison

   Choose task metrics, guardrails, eligible traffic, and assignment unit before exposure.

3. 3

   Run offline and shadow checks

   Reject obvious regressions before live impact and suppress shadow side effects.

4. 4

   Assign live cohorts

   Keep a user or workflow on a stable version according to the experiment design.

5. 5

   Analyze outcomes

   Compare quality, cost, latency, and subgroup behavior while accounting for incomplete observations.

6. 6

   Promote or revert

   Record the decision and retain the prior bundle and routing configuration for recovery.

Comparing resume-summary models

Recruiting operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

An internal assistant prepares factual summaries of candidate documents for recruiters.

The problem

A new model writes more concise summaries but may omit important experience details.

Agent requirement

Compare completeness and factual support while preserving a consistent recruiter workflow.

Architecture

A behavior registry and stable session assignment route eligible tasks to two evaluated bundles.

Runtime

• Reviewers score source-supported field coverage without seeing the bundle identity.
• The analysis checks omission rates by document format and language before promotion.

Failure scenario

If one language slice regresses despite a better overall score, the candidate is withheld or restricted to validated traffic.

Operational impact

Measure accepted summary quality, omissions, latency, and cost; user clicks alone can reflect presentation preference rather than factual correctness.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Using mutable model aliases as versions

Record resolved provider or artifact identity.

Promoting on average score alone

Inspect critical language, task, and tenant slices.

Calling a small canary a conclusive A/B test

Separate exposure control from statistical inference.

When to choose this pattern

Choose it when

• Use for model, prompt, adapter, or retrieval releases.
• Use when quality and cost tradeoffs need real workload evidence.
• Use when rollback must reproduce the previous behavior.

Avoid it when

• Avoid experiments without reliable task outcomes.
• Do not expose ineligible sensitive data to candidate providers.
• Avoid live effectful shadow runs.

Architecture checklist

• Register immutable behavior bundles.
• Choose a stable assignment unit.
• Predefine quality and operational gates.
• Suppress shadow side effects.
• Inspect critical subgroup regressions.
• Retain a tested rollback route.

Related concepts

• 31 — LLM Evaluation Frameworks (TruLens/Ragas)
• 32 — Model Bias Auditing
• 38 — Data Lineage Tracking
• 79 — Embedding Model Fine-Tuning
• 90 — LoRA / QLoRA Fine-Tuning
• 100 — Asynchronous Fine-Tuning Pipelines

Primary references

• MLflow Model Registry workflows