75 — Chunk Overlap Tuning

Concept overview

Chunk overlap repeats some source content across adjacent retrieval units so facts spanning a boundary remain recoverable. It is a hedge against imperfect segmentation, especially when paragraphs, definitions, or procedure steps extend across a fixed-size cut. Overlap is not a universal percentage: the right amount depends on document structure, tokenizer behavior, and retrieval tasks.

More overlap increases embedding and storage work and can make top results redundant. It may also bias ranking toward repeated text or inflate apparent evidence counts. Evaluate overlap jointly with chunk size, structural boundaries, parent expansion, and deduplication rather than optimizing it in isolation.

Why it matters in production

Proof of concept

A prototype tests several chunk-size and overlap pairs against questions whose answers cross boundaries.

Production

Production uses source offsets, stable chunk IDs, and duplicate-aware evaluation to distinguish real coverage from repeated passages.

Enterprise scale

At scale, tune by document family, estimate re-indexing costs, and track how overlap affects context tokens and retrieval diversity.

Architecture components

Structure-aware splitter

Prefers paragraphs, headings, and table boundaries where available.

Overlap policy

Defines token or structural overlap by document class.

Chunk lineage store

Preserves document revision and exact source ranges.

Duplicate-aware retriever

Recognizes overlapping hits before context assembly.

Boundary evaluation set

Tests whether answers survive segmentation without excessive duplication.

The architecture, mapped

flowchart LR
 D["Structured source document"] --> S["Boundary-aware splitter"]
 S --> A["Chunk A with shared tail"]
 S --> B["Chunk B with shared head"]
 A --> I["Vector index"]
 B --> I
 I --> R["Retrieved overlapping hits"]
 R --> U["Merge source ranges"]
 U --> C["Nonredundant evidence context"]

Runtime flow

1. 1

   Inspect source structure

   Identify the units that carry meaning, including headings, tables, and list conditions.

2. 2

   Select candidate settings

   Choose several chunk and overlap configurations within embedding limits.

3. 3

   Generate versioned chunks

   Record source offsets and revision identity for every chunk.

4. 4

   Evaluate boundary recall

   Test questions requiring information on both sides of a proposed cut.

5. 5

   Measure redundancy

   Count repeated source ranges in retrieved results and assembled context.

6. 6

   Deploy by document class

   Choose the smallest evaluated overlap that preserves needed evidence and retain a migration plan.

Retrieving specification conditions

Construction engineering · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A project assistant searches technical specifications containing requirements followed by exceptions on the next paragraph.

The problem

Fixed chunk boundaries separate a material requirement from its installation condition.

Agent requirement

Retrieve both parts while keeping duplicate specification text out of the answer context.

Architecture

A structure-aware splitter uses limited overlap and source-range deduplication.

Runtime

• A query matches the requirement near a boundary.
• The neighboring condition is present through overlap, and the context assembler merges shared ranges.

Failure scenario

If a large table exceeds chunk limits, a dedicated table strategy preserves headers instead of relying on overlap alone.

Operational impact

Measure boundary-question accuracy, duplicated context ratio, and total index size; overlap should solve a demonstrated segmentation failure.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Choosing overlap by habit

Tune against boundary-specific evaluation cases.

Measuring recall without deduplication

Count unique supporting source ranges.

Changing chunk settings without lineage

Version the chunking policy and rebuild consistently.

When to choose this pattern

Choose it when

• Use when important facts cross chunk boundaries.
• Use for imperfectly structured long documents.
• Use alongside source-offset deduplication.

Avoid it when

• Avoid large overlap for self-contained short records.
• Do not expect overlap to repair broken document extraction.
• Avoid optimizing overlap independently of chunk size.

Architecture checklist

• Create boundary-focused evaluation questions.
• Record source ranges and revisions.
• Test several chunk-size combinations.
• Measure duplicate context tokens.
• Preserve table headings explicitly.
• Version chunking policies before re-indexing.

Related concepts

• 64 — Parent-Child Chunking
• 65 — Re-ranking (Cohere/Cross-Encoders)
• 71 — Context Window Optimization
• 76 — Multi-Aspect Indexing
• 78 — Lost in the Middle Mitigation
• 92 — Pre-Computing Embeddings

Primary references

• Sentence Transformers: Retrieve and re-rank
• Lost in the Middle: How Language Models Use Long Contexts