A tool-using agent reads instructions from more places than the user. Documents, web pages, emails, database fields, and tool results can contain text that attempts to redirect the model. The architecture must assume those channels are untrusted.
Prompt injection is not solved by asking the model to ignore malicious instructions. The durable controls live outside the model: least privilege, explicit policy, data separation, confirmation, and limits on what any single run can do.
Map every instruction channel
List system instructions, user input, retrieved evidence, tool descriptions, tool output, memory, and messages from other agents. Mark which channels are trusted to define policy and which are only data. Preserve provenance when content is assembled.
Keep authorization deterministic
The model can request an action, but code should decide whether the current identity, task, resource, and parameters permit it. Scope credentials narrowly and issue them only at execution time. Do not expose secrets in context.
For irreversible or high-consequence actions, require confirmation through a trusted interface that shows the exact operation and affected resources.
Constrain tools and outputs
Use typed schemas, allowlists, resource limits, network boundaries, safe file locations, and idempotency. Treat model-generated code or queries as untrusted input and execute only inside an appropriate sandbox with explicit limits.
Sanitize output before it enters downstream interpreters. A model response should not become HTML, SQL, shell, or a financial instruction without validation designed for that context.
Evaluate indirect attacks
Place malicious and conflicting instructions inside realistic documents and tool responses. Test whether the agent reveals data, changes goals, invokes unauthorized tools, or conceals what happened. Repeat tests after model and prompt changes because attack resistance is behavioral.
Assume every external instruction is untrusted
Documents, web pages, messages, tickets, and tool outputs may contain language that attempts to redirect the agent. Mark data by origin and never let untrusted content redefine system policy, credentials, or tool authority. The model can classify or summarize that content, but deterministic controls must decide which actions are permitted.
Use capability-scoped credentials, destination allowlists, schema validation, rate limits, and transaction boundaries. Require approval when an operation crosses a consequence threshold, changes a sensitive record, or sends information outside its source boundary. These controls remain necessary even when the model appears resistant in prompt tests.
Test attacks as workflows
A prompt-injection suite should include indirect instructions in retrieved files, conflicting policy language, encoded requests, poisoned tool responses, and multi-step attempts to accumulate authority. Test whether the complete workflow leaks data or performs an action, not merely whether the model repeats a forbidden phrase.
Record which layer stopped the attempt. A robust system expects some model-level defenses to fail and relies on authorization, isolation, verification, and monitoring to limit impact. Convert successful attacks and near misses into regression cases, then rerun them whenever prompts, models, tools, or permissions change.
Executive decision record
The decision is what untrusted information the system may interpret and what authority it may exercise when that information attempts to redirect behavior. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on capability-scoped tools, deterministic policy enforcement, indirect-injection tests, data-flow review, approval boundaries, and attack regression suites. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is relying on prompt wording as the primary security boundary while the model holds broad credentials or can transmit sensitive data. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with the application and security owners jointly, because model behavior, tool design, identity, and incident response form one control system. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert capability-scoped tools, deterministic policy enforcement, indirect-injection tests, data-flow review, approval boundaries, and attack regression suites into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about what untrusted information the system may interpret and what authority it may exercise when that information attempts to redirect behavior. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by relying on prompt wording as the primary security boundary while the model holds broad credentials or can transmit sensitive data. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by the application and security owners jointly, because model behavior, tool design, identity, and incident response form one control system. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- Inventory trusted and untrusted instruction channels.
- Keep permissions and business rules outside prompts.
- Use scoped, short-lived credentials.
- Require approval for high-consequence operations.
- Validate tool arguments and downstream output.
- Limit actions, time, cost, and reachable resources.
- Maintain indirect prompt-injection regression tests.
Engagement scenario
An email-triage agent reads an attachment containing instructions to forward confidential messages. The attachment is labeled untrusted evidence, the agent lacks a forwarding credential, and the requested external recipient fails policy validation. The run is denied and added to the security evaluation set.
Continue reading
- [Enterprise agent architecture](/enterprise-ai-agent-architecture)
- [Human-in-the-loop agent design](/human-in-the-loop-ai-agent-design)
Sources and further reading
- [OWASP Top 10 for LLM applications](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)
- [Anthropic: Trustworthy agents in practice](https://www.anthropic.com/research/trustworthy-agents)
- [NIST Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)