Five principles that contain a compromised agent
An agent that gets tricked doesn't just produce a bad sentence — it can call a tool. Design for what it's able to do once that happens, not just for stopping the trick.
By Piotr Zenon Żabrowski · Published · Updated
01Stop on uncertainty
A bad retrieval, a contradictory tool result, or a step with no clear justification should halt the agent and ask for a decision. An agent that always answers has no failure mode except acting wrong with confidence.
02Keep data out of the instruction channel
Only the developer's system prompt and the user's request carry authority over what the agent does next. A fetched page or tool output is content to weigh, not a command to follow — blurring that line is how indirect prompt injection turns a lookup into an attacker's order.
03Least privilege per tool
A search tool doesn't get filesystem access; a scheduler doesn't get billing access. Partition memory by agent and process so a compromised run can't read another's state, and scope each tool's permissions to the minimum data and operations it needs — memory isolation and tool authorization are separate controls, not one.
04Route risky actions through a person
Sending a message, deleting a file, moving money, or running code needs a specific confirmation, gated by what the action does — read, modify, delete, or send externally. A single generic approval prompt just trains people to click through it.
05Record enough to reconstruct a run
Every tool call, its parameters, what came back, and what reached the user needs a record. Without that trail, figuring out what a compromised agent touched is guesswork after the fact.
One control sits above all five: a user-accessible way to halt a running agent process, which the CNIL's technical guidance on agent-based AI calls for directly. The five principles bound what an agent can do at each step, but none stops a loop already running.
Before an agent gets write access to anything
- Each tool uses separate credentials scoped to the minimum systems, data, and operations it needs.
- Every irreversible action has a confirmation step classified by what it does.
- Treat retrieved content and tool output as untrusted data, and enforce allowed actions in the orchestrator rather than relying on the model to obey labels.
- Tool calls, parameters, and results are logged well enough to reconstruct a bad run.
- The agent has an explicit stop condition for low-confidence or contradictory input.
- A person can halt the running process on demand.
Frequently asked questions
Do these five principles replace threat modeling for the rest of the system?
No. They address one failure mode: an agent that has already been fooled or compromised. The model and retrieval layer still need their own controls; these five bound what a compromised agent's tool access lets it do next.
Isn't a kill switch just an operational nicety?
The five principles limit what an agent can do at each step, but none stop a live process mid-loop. A user-accessible halt is the control that does, and the CNIL's technical guidance calls for exactly that: a way for the user to stop a running agent.
Related
AI security
How to secure GenAI, LLM and agentic AI systems: current OWASP threat classes, layered controls, agent authority, retrieval isolation and monitoring.
AI security assessment
Security assessment of GenAI and LLM systems: prompt injection, tool and agent abuse, data leakage, model supply chain and monitoring gaps.
Agentic threat models
Why the shift from generative query-response to an autonomous agentic loop changes the threat model of an AI system from bad output to unintended action.
Agent blast radius
A bad model decision becomes an incident through tool calls. Agent memory determines how long and how far the damage can spread.
What contains a CLI agent
Four failure modes, the profile that answers them, and why enforcement can't live inside the agent's own instructions.
Where the credential lives
Two shapes of agent isolation, the one that keeps keys out of the process, and the failure both shapes still miss.