Agentic Security & Safety
How do you prevent prompt injection in tool-use agents?
QUICK ANSWER
Prevent prompt injection in tool-use agents by validating and sanitizing all untrusted user and tool outputs, enforcing strict schema validation, isolating tool execution from core agent instructions, and using system prompt instructions that warn the model to treat tool data strictly as untrusted content.
Prompt Injection Mitigation Playbook
- Context Segregation: Demarcate untrusted data using clear structural boundaries (e.g. XML tags like
<untrusted_tool_output>) in messages sent to the model. - Defensive Instruction Design: Provide instructions in the system prompt explicitly telling the agent to ignore any model command directives embedded inside tool results or third-party content.
- Input/Output Schema Enforcement: Validate arguments passed by the model against a strict JSON schema before execution, rejecting any code-injection patterns.
- Privilege Minimization: Grant execution tools the minimum necessary system access (e.g. read-only permissions for public files).
Verified against: Anthropic Safety & Guardrails Specification