87 — Batch Prompting

Concept overview

Batch prompting places several logically separate items into one model request and asks for an item-indexed response. It can amortize repeated instructions for small homogeneous tasks, but it shares context and failure fate across the items. A malformed response, truncated output, or adversarial item can affect the whole batch.

This differs from server-side inference batching, where an engine schedules independent requests together, and from asynchronous batch APIs, which submit many independent requests for deferred processing. Those mechanisms need not combine user data into one prompt. Architects should choose the appropriate batching layer before accepting the isolation and reliability costs of prompt-level packing.

Why it matters in production

Proof of concept

A prototype compares one-item calls with small homogeneous prompt batches and checks item-level accuracy.

Production

Production assigns stable item IDs, caps input and output budgets, and validates that every expected item appears exactly once.

Enterprise scale

At scale, separate tenant and policy scopes, use adaptive batch sizing, and retry only failed items while monitoring head-of-line delay and cross-item contamination.

Architecture components

Batch builder

Groups compatible items under shared task, policy, and deadline constraints.

Prompt envelope

Separates item data and specifies an item-indexed output schema.

Token estimator

Bounds total input and expected output before dispatch.

Result demultiplexer

Maps model outputs back to stable item IDs and detects omissions or duplicates.

Retry queue

Resubmits failed items independently without repeating accepted results.

The architecture, mapped

flowchart LR
 I["Compatible input items"] --> B["Bounded batch builder"]
 B --> P["One item-indexed prompt"]
 P --> M["Model response"]
 M --> D["Demultiplex and validate IDs"]
 D --> S["Accepted item results"]
 D --> R["Retry missing or invalid items"]
 R --> B

Runtime flow

1. 1

   Choose the batching layer

   Decide whether prompt packing, engine batching, or deferred independent requests fit the workload.

2. 2

   Group compatible items

   Require the same task contract and permitted data-sharing scope.

3. 3

   Assign identities and budget

   Give every item a stable ID and reserve sufficient output capacity.

4. 4

   Execute the batch

   Submit the bounded prompt and retain the batch-to-item mapping.

5. 5

   Validate each result

   Check schema, coverage, duplicate IDs, and task acceptance per item.

6. 6

   Retry selectively

   Split or resubmit only missing or failed items with preserved identity.

Tagging a catalog of short descriptions

Media operations · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A media team classifies thousands of short program descriptions into a stable editorial taxonomy.

The problem

Single-item requests repeat long instructions, while oversized batches omit items near the end.

Agent requirement

Reduce repeated prompt overhead without losing item-level accountability.

Architecture

The batch builder groups similar short descriptions and requires one structured result per catalog ID.

Runtime

• A bounded group is sent with a fixed taxonomy and explicit IDs.
• The demultiplexer accepts valid tags and retries only omitted descriptions.

Failure scenario

If one item contains instruction-like text, it remains data and cannot alter the batch taxonomy; contaminated results are rejected.

Operational impact

Measure cost per accepted item, omission rate, batch latency, and contamination; aggregate response success does not mean every catalog record was processed.

Architecture tradeoffs

Failure scenarios & recovery

Common architecture mistakes

Calling all batching prompt packing

Distinguish prompt, engine, and asynchronous API layers.

Mixing unrelated tenants

Enforce a common authorized data scope.

Retrying an entire accepted batch

Track and retry only unsuccessful items.

When to choose this pattern

Choose it when

• Use for small homogeneous classification or extraction items.
• Use when shared instructions dominate request cost.
• Use when per-item validation is reliable.

Avoid it when

• Avoid prompt packing across incompatible privacy scopes.
• Avoid large variable-length outputs with unpredictable budgets.
• Do not use it for items requiring independent conversational reasoning.

Architecture checklist

• Choose the correct batching layer.
• Assign stable item IDs.
• Cap input and output tokens.
• Validate exact item coverage.
• Test cross-item contamination.
• Retry failed items independently.

Related concepts

• 6 — Asynchronous Agent Execution
• 41 — Idempotent Tool Execution
• 58 — Partial Success Management
• 88 — vLLM / TensorRT-LLM Engines
• 89 — Serverless LLM Architecture
• 94 — Token Cost Allocation Tracking

Primary references

• NVIDIA Triton: TensorRT-LLM backend