Enterprise-Grade Security & Governance

24 — Prompt Injection Mitigation

Constrain untrusted instructions at data boundaries and prevent retrieved content from acquiring tool authority, secrets, or control over the agent workflow.

Concept overview

Prompt injection occurs when untrusted content attempts to redirect a model away from its intended task or policy. The content may arrive in a user message, retrieved page, document, image, or tool result. Because instructions and data share a model context, delimiters and warnings alone cannot establish a complete security boundary.

Design around consequences. Keep secrets out of model context, constrain tool capabilities, validate outputs, and enforce authorization outside the model. A retrieved document may contain useful facts and malicious instructions simultaneously; the architecture must preserve evidence utility without treating the document as a source of authority.

Why it matters in production

Proof of concept

A prototype should include malicious documents and tool responses in its test corpus, even when all tools are read-only.

Production

Production separates trusted configuration from retrieved data, limits outbound destinations, scopes credentials, and requires confirmation for specified consequential actions. Detectors provide signals rather than guarantees.

Enterprise scale

At scale, maintain adversarial regression suites, provenance-aware traces, and incident containment. Evaluate successful unauthorized effects and data exposure, not merely whether the model repeated suspicious text.

Architecture components

Trust-aware prompt assembler

Labels source provenance and prevents external text from replacing trusted configuration.

Capability gateway

Enforces narrow tool permissions and rejects unauthorized operations independently of model output.

Content detector

Flags suspicious instruction patterns and risky payloads for policy handling.

Egress controller

Restricts destinations, data classes, and request shapes that tools can transmit.

Adversarial evaluator

Tests indirect injection paths and verifies actual boundary enforcement.

The architecture, mapped

Untrusted evidence cannot authorize tools — Prompt handling reduces exposure, while the capability gateway prevents unauthorized effects.

flowchart LR
 D["External document"] --> T["Provenance and inspection"]
 T --> M["Model context"]
 C["Trusted task configuration"] --> M
 M --> G["Independent capability gateway"]
 P["Authorization and egress policy"] --> G
 G --> A["Approved business API"]
 G --> X["Denied action and audit event"]
 M --> V["Output validation"]

Contain a malicious retrieval result — The control remains effective even when the model proposes the attacker-requested action.

sequenceDiagram
 participant D as External document
 participant M as Model
 participant G as Tool gateway
 participant P as Egress policy
 participant A as Audit record
 D->>M: Evidence containing hostile instructions
 M->>G: Proposed upload of internal notes
 G->>P: Check destination and data authority
 P-->>G: Denied outside approved scope
 G-->>M: Unauthorized action rejected
 G->>A: Record source provenance and denied effect

Runtime flow

1
Classify sources
Mark user input, retrieved evidence, tool output, and trusted configuration by provenance.
2
Minimize exposure
Remove unnecessary secrets and limit the data and capabilities available for the task.
3
Inspect content
Apply detection and normalization while retaining the distinction between evidence and instructions.
4
Generate proposal
Allow the model to use evidence but require structured action requests.
5
Enforce boundaries
Validate authorization, arguments, destinations, and approval requirements before any effect.
6
Verify outcome
Check public output and audit traces for leakage or unauthorized actions, then quarantine compromised sources.

Reading supplier proposals containing hostile instructions

Enterprise procurement · Illustrative enterprise scenario, not a reported customer outcome.

Business situation

A procurement assistant reviews supplier PDFs and can retrieve internal evaluation templates.

The problem

A supplier inserts text instructing the assistant to upload evaluation notes to an external endpoint disguised as a verification service.

Agent requirement

The assistant must extract proposal facts without allowing the supplier document to control internal tools or data sharing.

Architecture

Documents enter as untrusted evidence; the agent has no arbitrary network tool, and outbound actions pass through an allowlisted gateway.

Runtime

The PDF is parsed with source provenance, and suspicious instruction content is recorded for review.
The model's attempted external upload is rejected because neither the task nor the caller authorizes that destination or data transfer.

Failure scenario

If a detector misses the payload, capability and egress controls still prevent the upload; the proposal can be quarantined based on the denied action.

Operational impact

Measure unauthorized-action success, sensitive-data exposure, benign-document false positives, and containment time across realistic attack variants.

Architecture tradeoffs

Decision	Benefit	Cost
Narrow capabilities	Injection has fewer harmful actions available.	Some useful flexible workflows require additional adapters.
Content detection	Known attack patterns can be flagged early.	Obfuscation and novel payloads can evade classifiers.
Strict egress policy	Exfiltration paths are reduced independently of model behavior.	Integrations need deliberate destination and data approvals.

Failure scenarios & recovery

Malicious text is hidden in a tool result

Detect: Provenance trace shows the instruction originated outside trusted configuration.

Recover: Reject unauthorized actions and isolate the affected source.

A safety classifier approves an attack

Detect: Adversarial tests still observe an attempted prohibited effect.

Recover: Strengthen the execution boundary instead of trusting classifier confidence.

Common architecture mistakes

Treating delimiters as a security boundary

Enforce authority in tools and data services.

Putting reusable secrets in prompts

Use scoped credentials outside model-visible context.

Testing only direct user attacks

Include retrieval, files, images, and tool-response injection.

When to choose this pattern

Choose it when

Agents consume content from users or external systems.
Tools can access private data or perform actions.
Retrieved evidence shares context with trusted instructions.

Avoid it when

No model or instruction-following component processes the content.
A proposed control relies solely on perfect attack detection.
The workflow requires unrestricted secret-bearing execution.

Architecture checklist

Map every untrusted content entry point.
Keep credentials outside model context.
Restrict tool scope and outbound destinations.
Test indirect injection with realistic artifacts.
Measure unauthorized effects rather than refusals alone.
Define quarantine and incident-response paths.

Primary references

OWASP: LLM prompt injection prevention