38 — Data Lineage Tracking

Concept overview

Data lineage records relationships between inputs, transformations, execution runs, and outputs. For agent systems, useful lineage extends beyond ETL tables to document versions, chunks, embeddings, retrieval sets, prompts, model configurations, tool observations, and published artifacts. Trace spans describe execution timing; lineage explains derivation and dependency.

Choose stable identifiers and appropriate granularity. A document-level link may be enough for discovery but insufficient to explain which passage supported a claim. Store references and transformation metadata rather than duplicating sensitive content in the lineage catalog. Lineage supports investigation and impact analysis; it does not prove that a transformation was correct.

Why it matters in production

Proof of concept

A prototype can retain source identifiers and model versions beside generated outputs. Add transformation identities before rebuilding indexes or changing chunking logic.

Production

Production emits lineage events for ingestion and generation, preserves source versions, and links outputs to retrieval evidence and configuration. Handle deletions and revoked access across derived artifacts.

Enterprise scale

At scale, lineage becomes a graph supporting impact queries and selective reprocessing. Avoid unbounded event cardinality by choosing meaningful artifact boundaries and retention policies.

Architecture components

Artifact identity

Assigns stable names and versions to sources, chunks, embeddings, and generated outputs.

Transformation recorder

Captures job, run, code, configuration, and input-output relationships.

Lineage catalog

Stores derivation edges and supports upstream and downstream impact queries.

Generation evidence link

Connects answers to retrieved passages, tool results, and model configuration.

Change propagator

Uses lineage to identify artifacts requiring invalidation, deletion, or recomputation.

The architecture, mapped

flowchart LR
 S["Source document version"] --> P["Parser run"]
 P --> C["Chunk versions"]
 C --> E["Embedding run"]
 E --> I["Index snapshot"]
 I --> R["Retrieved evidence set"]
 R --> G["Generation run"]
 M["Model and prompt versions"] --> G
 G --> A["Published answer"]
 S --> L["Lineage catalog"]
 G --> L

Runtime flow

1. 1

   Identify source version

   Record the authoritative object, revision, ownership, and relevant access metadata.

2. 2

   Record transformation

   Emit input-output relationships for parsing, chunking, embedding, and indexing runs.

3. 3

   Capture retrieval set

   Associate the generation run with the actual evidence used, not every possible source.

4. 4

   Link output

   Store model and prompt versions with the published artifact and claim evidence references.

5. 5

   Query impact

   Traverse dependencies when a source, policy, or transformation changes.

6. 6

   Apply remediation

   Rebuild or invalidate affected artifacts and record the corrective transformation history.

Tracing an assistant answer to an operating procedure

Pharmaceutical manufacturing · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A manufacturing support assistant answers questions about approved equipment procedures across several facilities.

The problem

When a procedure revision is withdrawn, the team cannot identify which indexed chunks and generated guidance used it.

Agent requirement

The assistant must support source-version traceability and targeted invalidation of affected content.

Architecture

Ingestion records document, parser, chunk, and embedding versions; generation records the retrieved evidence set and output artifact.

Runtime

• An operator asks about cleaning setup, and the answer links to the exact approved procedure revision used.
• When that revision is superseded, a lineage query identifies affected index entries and retained answer artifacts for review or invalidation.

Failure scenario

If a pipeline omits lineage events, its outputs remain quarantined from the controlled corpus until provenance is repaired.

Operational impact

Measure lineage completeness, impact-query coverage, stale-artifact removal time, and reproducibility of sampled answers. Traceability does not replace procedure approval.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Confusing traces with complete lineage

Record durable input-output relationships and source versions.

Linking answers to the whole corpus

Capture the actual retrieved and used evidence set.

Ignoring configuration dependencies

Include chunking, embedding, prompt, and model versions.

When to choose this pattern

Choose it when

• Sources and transformations change over time.
• Incidents require tracing affected generated artifacts.
• Deletion and reprocessing need precise dependency information.

Avoid it when

• Outputs are disposable and have no retained dependencies.
• Identifiers cannot remain stable across transformations.
• The catalog would collect sensitive payloads without a clear purpose.

Architecture checklist

• Define artifact identity and version rules.
• Record every material transformation boundary.
• Link answers to actual evidence sets.
• Include model and prompt configuration.
• Test deletion and invalidation propagation.
• Monitor missing lineage events.

Related concepts

• 27 — Responsible AI Governance (ISO/IEC 42001)
• 28 — Deterministic Data Grounding
• 30 — Non-Repudiation Logs
• 64 — Parent-Child Chunking
• 77 — Real-time Vector Ingestion

Primary references

• OpenLineage object model