80 — Token Pruning

Concept overview

Token pruning reduces the text sent to a model by selecting or compressing information that contributes to the task. It can operate at document, sentence, or token level. Learned compressors estimate importance differently from simple truncation, but all pruning methods risk removing a condition or relationship the final answer needs.

Prompt-token pruning is distinct from pruning model weights or evicting attention-cache entries. Its success criterion is end-to-end task quality at a lower total cost, including the compressor itself. Exact quotations, code, structured records, numbers, and contractual language often require protected spans rather than aggressive token deletion.

Why it matters in production

Proof of concept

A prototype compares sentence selection, simple truncation, and a learned compressor on the same tasks.

Production

Production marks protected spans, retains source alignment, and evaluates factual and structural preservation at several compression ratios.

Enterprise scale

Enterprise scale routes only suitable workloads through compression and monitors quality drift, compressor latency, and total inference savings rather than token reduction alone.

Architecture components

Budget controller

Sets a target based on available context and the task’s tolerated information loss.

Importance scorer

Ranks passages or tokens using a documented selection method.

Protected-span policy

Preserves instructions, identifiers, negation, units, and exact-evidence requirements.

Compression stage

Produces reduced context with links to original source ranges.

Quality gate

Checks required facts and compares downstream performance with an uncompressed baseline.

The architecture, mapped

flowchart LR
 C["Candidate context"] --> P["Protected-span marking"]
 P --> S["Importance scoring"]
 S --> R["Budgeted pruning"]
 R --> V["Fact and structure checks"]
 V --> M["Reduced model request"]
 V --> F["Fallback to original or smaller source set"]

Runtime flow

1. 1

   Classify the task

   Determine whether the task permits lossy context or requires exact source wording.

2. 2

   Mark protected information

   Identify mandatory instructions, numbers, entity names, and evidence-critical conditions.

3. 3

   Select a compression level

   Choose an evaluated ratio that meets the latency and quality budget.

4. 4

   Prune with provenance

   Remove low-value content while retaining source mapping and protected spans.

5. 5

   Validate the result

   Check required fields, negation, and structure before sending the compressed prompt.

6. 6

   Measure total benefit

   Include compression overhead, generation cost, and error rates in the acceptance decision.

Summarizing repetitive contact logs

Customer service analytics · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A service-quality agent reviews long contact logs containing repeated greetings, routing text, and issue updates.

The problem

Boilerplate consumes most of the prompt while a few resolution facts matter.

Agent requirement

Reduce context cost without losing the customer’s issue, promised action, or unresolved status.

Architecture

A compressor removes repetitive low-value spans and protects commitments, dates, and final disposition.

Runtime

• The pipeline marks promised follow-up and resolution statements.
• It compresses repeated routing chatter and checks that protected facts remain before summarization.

Failure scenario

If pruning removes a negation that changes whether an issue was resolved, validation rejects the compressed version.

Operational impact

Measure commitment-retention accuracy and total processing cost; a smaller prompt is not an improvement when it changes the operational meaning.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Optimizing compression ratio alone

Measure task quality and total latency.

Pruning mandatory instructions

Reserve protected policy and output-contract segments.

Losing source alignment

Retain original ranges so citations remain verifiable.

When to choose this pattern

Choose it when

• Use for repetitive long context with clear evidence priorities.
• Use when prefill cost is material.
• Use when quality can be evaluated against an uncompressed baseline.

Avoid it when

• Avoid exact quotation or code-transformation tasks without protected structure.
• Do not compress already short requests by default.
• Avoid pruning when required evidence cannot be identified.

Architecture checklist

• Distinguish prompt pruning from model compression.
• Define protected spans.
• Evaluate several compression levels.
• Preserve source alignment.
• Test negation and numeric fidelity.
• Measure compressor overhead and downstream errors.

Related concepts

• 19 — Context Condensation
• 70 — Dynamic Prompt Assembly
• 71 — Context Window Optimization
• 75 — Chunk Overlap Tuning
• 78 — Lost in the Middle Mitigation
• 84 — Model Quantization (AWQ/GGUF)

Primary references

• LLMLingua: Compressing Prompts for Accelerated Inference