69 — Vector Database Scaling (Pinecone/Milvus)

Concept overview

Vector database scaling is a capacity and correctness problem across several dimensions: vector count, dimensionality, metadata size, query rate, update rate, and filter selectivity. Approximate nearest-neighbor indexes trade search work and memory for recall. Adding replicas may increase read capacity, while sharding changes data distribution and query fan-out.

Managed and self-operated systems expose different operational boundaries, so capacity planning must use the selected product’s behavior rather than assume interchangeable guarantees. Consistency, index readiness, deletion visibility, and metadata filtering affect application correctness. A successful upsert acknowledgment does not necessarily mean every subsequent search immediately sees the change.

Why it matters in production

Proof of concept

A prototype uses representative vectors and metadata rather than a tiny unfiltered sample.

Production

Production benchmarks recall and tail latency under concurrent reads, writes, and selective filters, then sets capacity and freshness objectives.

Enterprise scale

Enterprise scale introduces tenant placement, hot-partition handling, rebuild planning, disaster recovery, and cost models covering replicas, storage, and re-embedding.

Architecture components

Ingestion workers

Validate vectors and metadata and write versioned records idempotently.

Index partitions

Distribute vector and scalar data across storage and compute boundaries.

Query replicas

Provide search capacity with explicit health and index readiness.

Routing layer

Selects tenant partitions and merges results across required shards.

Capacity telemetry

Tracks latency, recall samples, memory, ingestion lag, and hot partitions.

The architecture, mapped

flowchart LR
 S["Versioned source changes"] --> I["Ingestion workers"]
 I --> A["Partition A"]
 I --> B["Partition B"]
 Q["Scoped query"] --> R["Tenant and shard router"]
 R --> X["Read replica A"]
 R --> Y["Read replica B"]
 A --> X
 B --> Y
 X --> M["Merge candidates"]
 Y --> M

Runtime flow

1. 1

   Profile the workload

   Measure vector sizes, metadata cardinality, query shapes, and update frequency.

2. 2

   Choose partitioning

   Align tenant isolation and expected access patterns while avoiding oversized hotspots.

3. 3

   Build and validate indexes

   Check dimensions, metric choice, and recall against an exact-search sample.

4. 4

   Serve constrained queries

   Route to relevant partitions and apply supported filters with bounded fan-out.

5. 5

   Scale the bottleneck

   Add read replicas, rebalance partitions, or tune indexes according to observed pressure.

6. 6

   Rebuild safely

   Create a new index version, validate it, and switch routing with a rollback path.

Scaling a multilingual product knowledge index

E-commerce · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A marketplace assistant searches product specifications across many sellers and frequent catalog updates.

The problem

A few large sellers create hot partitions while promotions increase query volume and update traffic.

Agent requirement

Maintain scoped retrieval quality without letting one seller consume the whole search tier.

Architecture

Tenant-aware routing combines partition limits, read replicas, and versioned ingestion checkpoints.

Runtime

• A query targets only authorized seller partitions.
• The platform monitors selective-filter latency and splits an oversized partition after validating the new placement.

Failure scenario

If a new index revision lags or loses recall, routing returns to the previous validated revision.

Operational impact

Measure recall, p95 search latency, index visibility lag, and cost per workload class; raw vector count does not determine required capacity alone.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Benchmarking only unfiltered queries

Include real ACL and metadata predicates.

Scaling replicas to fix ingestion bottlenecks

Identify the constrained subsystem first.

Migrating embeddings in place

Build and validate a parallel index before switching.

When to choose this pattern

Choose it when

• Use for growing corpora or sustained concurrent retrieval.
• Use when tenant skew affects latency.
• Use when updates and queries need separate capacity planning.

Avoid it when

• Avoid distributed indexing for a small stable corpus without need.
• Do not assume immediate read visibility after writes.
• Do not trade away recall without downstream evaluation.

Architecture checklist

• Benchmark representative filtered workloads.
• Measure recall against an exact baseline.
• Document consistency and visibility behavior.
• Plan hot-tenant limits.
• Version embedding and index schemas.
• Exercise rebuild and restore procedures.

Related concepts

• 62 — Hybrid Search (Dense + Sparse)
• 63 — Metadata Pre-Filtering
• 68 — Time-Weighted Vector Retrieval
• 77 — Real-time Vector Ingestion
• 79 — Embedding Model Fine-Tuning
• 92 — Pre-Computing Embeddings

Primary references

• Milvus: Filtered search
• Milvus: Multi-vector hybrid search
• Milvus: Consistency levels