77 — Real-time Vector Ingestion

Concept overview

Real-time vector ingestion moves source changes into a searchable index with a defined freshness objective. It usually combines change data capture or application events with parsing, chunking, embedding, and index writes. Real-time is a latency target across this pipeline, not a guarantee that an acknowledged event is immediately visible to every query.

The difficult cases are ordering, duplicate delivery, deletes, permission changes, and reprocessing after a model upgrade. Stable document versions and idempotent writes prevent old events from replacing newer evidence. Tombstones and authorization updates deserve the same reliability as content additions because stale access can be more serious than missing search results.

Why it matters in production

Proof of concept

A prototype measures elapsed time from source commit to successful retrieval for a changed record.

Production

Production adds durable offsets, deduplication, version checks, dead-letter handling, and explicit deletion propagation.

Enterprise scale

Enterprise scale separates backfills from live updates, monitors partition lag, and supports index migrations without starving current changes.

Architecture components

Change capture source

Emits committed content, deletion, and permission changes with stable identity.

Durable event log

Buffers updates and preserves replay positions.

Transformation workers

Parse and chunk documents using a versioned policy.

Embedding and index writers

Create compatible vectors and apply idempotent versioned updates.

Freshness verifier

Checks when a committed source revision becomes searchable and old revisions disappear.

The architecture, mapped

flowchart LR
 S["Source transaction"] --> C["Change capture"]
 C --> L["Durable event log"]
 L --> T["Versioned chunk transformation"]
 T --> E["Embedding workers"]
 E --> V["Idempotent vector writes"]
 V --> F["Search visibility probe"]
 L --> D["Delete and permission events"]
 D --> V

sequenceDiagram
 participant L as Change log
 participant W as Index worker
 participant V as Vector store
 participant F as Freshness monitor
 L->>W: Document revision event
 W->>V: Read applied revision
 V-->>W: Current revision
 W->>W: Reject stale event or transform new revision
 W->>V: Apply idempotent upsert or tombstone
 V-->>W: Write acknowledgment
 W->>L: Commit processing position
 F->>V: Probe search visibility

Runtime flow

1. 1

   Capture committed change

   Record source identity, revision, operation type, and ordering information.

2. 2

   Validate and transform

   Check schema and access metadata before producing versioned chunks.

3. 3

   Embed changed content

   Reuse unchanged chunk hashes and generate vectors only for required updates.

4. 4

   Apply index mutation

   Upsert new revisions and remove superseded chunks with conditional ordering safeguards.

5. 5

   Commit processing position

   Advance the durable offset only after the required index operation succeeds.

6. 6

   Verify visibility

   Probe retrieval freshness and deletion propagation separately from write acknowledgment.

Updating outage knowledge during restoration

Telecom operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A network assistant answers internal questions using incident tickets and restoration notes.

The problem

Batch indexing leaves resolved incidents appearing active and misses newly identified affected areas.

Agent requirement

Make committed updates searchable within a defined freshness target and remove superseded status.

Architecture

A change stream feeds version-aware chunking and embedding workers with a dedicated live-update queue.

Runtime

• An incident status changes and emits a new source revision.
• The index writer replaces old chunks and a probe confirms that current status is retrievable.

Failure scenario

A delayed older event is rejected by revision comparison instead of reopening the incident in the index.

Operational impact

Track source-to-search latency, deletion lag, and failed-event age; queue throughput alone does not prove current answers.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Calling write acknowledgment real-time visibility

Measure query-visible revision lag.

Ignoring out-of-order updates

Use source versions and conditional writes.

Rebuilding all embeddings for every edit

Hash chunks and reuse unchanged representations.

When to choose this pattern

Choose it when

• Use for frequently changing operational knowledge.
• Use when stale status causes incorrect agent actions.
• Use when deletion and permission propagation have explicit objectives.

Avoid it when

• Avoid streaming complexity for rarely changed small corpora.
• Do not promise instantaneous consistency without store support.
• Avoid live pipelines without replayable event history.

Architecture checklist

• Define source-to-search freshness targets.
• Persist replay positions durably.
• Handle duplicates and old revisions.
• Propagate deletes and ACL changes.
• Isolate backfills from live traffic.
• Probe actual search visibility.

Related concepts

• 15 — Event-Driven Agent Triggers
• 38 — Data Lineage Tracking
• 41 — Idempotent Tool Execution
• 48 — Dead-Letter Queues (DLQ) for Agents
• 68 — Time-Weighted Vector Retrieval
• 69 — Vector Database Scaling (Pinecone/Milvus)

Primary references

• Debezium architecture
• Milvus consistency