65 — Re-ranking (Cohere/Cross-Encoders)

Concept overview

Re-ranking is a second retrieval stage that scores candidate passages against the query after an inexpensive first-stage search. A cross-encoder jointly processes query and passage, allowing richer interactions than separately embedded vectors. Hosted reranking services provide a similar architectural role, with service-specific limits and contracts.

The reranker cannot recover a relevant document absent from the candidate set. First-stage recall and second-stage ordering must therefore be measured separately. Its score is a ranking signal, not a calibrated probability that a passage is correct or authorized. Long passage truncation and unsupported languages can distort ranking even when the service returns successfully.

Why it matters in production

Proof of concept

Start with a fixed retrieval shortlist and labeled relevance judgments to isolate whether reranking improves order.

Production

Production budgets candidate count and passage length, sets timeouts, and defines an acceptable fallback to first-stage ranking.

Enterprise scale

At scale, batch compatible scoring work, monitor language and domain slices, and version reranker changes alongside retrieval evaluations.

Architecture components

Candidate retriever

Produces a diverse high-recall shortlist under trusted filters.

Passage normalizer

Preserves title, relevant text, and provenance within reranker limits.

Cross-encoder service

Scores each query-passage pair using joint attention or an equivalent ranking model.

Selection policy

Chooses top evidence while controlling redundancy and source diversity.

Ranking evaluator

Tracks recall, ordering quality, latency, and downstream answer support.

The architecture, mapped

flowchart LR
 Q["Question"] --> H["Hybrid candidate retrieval"]
 H --> N["Bounded passage pairs"]
 N --> X["Cross-encoder scoring"]
 X --> S["Relevance and diversity selection"]
 S --> C["Cited generation context"]
 X --> F["Timeout fallback to evaluated ranking"]

Runtime flow

1. 1

   Retrieve candidates

   Collect a bounded shortlist from dense, sparse, or hybrid retrieval.

2. 2

   Prepare pairs

   Attach the question to each passage and preserve essential headings and source identity.

3. 3

   Score relevance

   Run the reranker under a deadline and record its model version.

4. 4

   Select evidence

   Sort by relevance while removing duplicates and respecting document diversity requirements.

5. 5

   Apply answerability rule

   Do not force an answer when even the best candidates lack sufficient evidence.

6. 6

   Record outcome

   Compare accepted evidence and eventual answer support against labeled evaluation examples.

Finding the relevant experimental protocol

Pharmaceutical research · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A research assistant searches approved laboratory protocols and historical method notes.

The problem

Many passages share assay terminology, but only a few address the requested sample preparation conditions.

Agent requirement

Rank passages by the specific experimental question before drafting an evidence-linked summary.

Architecture

Hybrid retrieval supplies candidates and a domain-evaluated cross-encoder selects the most relevant sections.

Runtime

• The query retrieves passages mentioning the assay family.
• Reranking promotes the protocol containing the requested sample condition and demotes unrelated validation notes.

Failure scenario

If the reranker times out, the service uses a tested retrieval-only mode and exposes reduced evidence confidence where appropriate.

Operational impact

Track relevant-passage rank and unsupported-answer rate alongside added latency; fluent summaries do not prove that ranking improved.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Expecting reranking to fix missing candidates

Improve first-stage recall before adjusting the reranker.

Interpreting scores as factual confidence

Validate evidence support independently.

Ignoring language-specific quality

Evaluate each supported language and domain slice.

When to choose this pattern

Choose it when

• Use when retrieved candidates are relevant but poorly ordered.
• Use before expensive generation over a limited context.
• Use for queries requiring fine distinctions among similar passages.

Avoid it when

• Avoid when first-stage recall is the dominant failure.
• Avoid unbounded reranking of the whole corpus.
• Do not rerank unauthorized candidates as a substitute for filtering.

Architecture checklist

• Measure first-stage recall separately.
• Set candidate and token budgets.
• Version the ranking model.
• Test truncation behavior.
• Evaluate fallback ranking quality.
• Track diversity and answer-support metrics.

Related concepts

• 31 — LLM Evaluation Frameworks (TruLens/Ragas)
• 62 — Hybrid Search (Dense + Sparse)
• 64 — Parent-Child Chunking
• 66 — Query Deconstruction / Expansion
• 71 — Context Window Optimization
• 79 — Embedding Model Fine-Tuning

Primary references

• Sentence Transformers: Retrieve and re-rank