Three layers, three risk profiles
"We reviewed the AI system" is not a scope. A model, the service wrapped around it, and an agent acting through that service fail in different ways and need different questions.
By Piotr Zenon Żabrowski · Published · Updated
Same weights, three different attack surfaces
Each layer draws its own trust boundary, and conflating them means applying the wrong control. At the innermost boundary, trained weights take tokens in and push tokens out; no session state, no outbound network access, so exposure is limited to whatever leaked in during training. Wrap that in an API, a queue, and stored conversation history, and exposure shifts to retention and cross-tenant isolation. Wrap it again in an orchestrator that lets those weights call tools on a self-built plan, and exposure shifts again: a bad tool call can run with the wrong permissions or act on an instruction nobody typed. Same weights, three boundaries, three control sets — scoping a review means naming one.
Why the layer changes the questions
| Layer | What can go wrong | What to ask a vendor |
|---|---|---|
| Model | Training data memorized and later reproduced; no built-in fact-checking | What data trained this, and has extraction risk been tested? |
| Service/application | Logging, retention, and cross-tenant exposure at the API and UI boundary | What is logged, where is it stored, and for how long? |
| Agent | A tool call executes with the wrong permissions, or acts on a poisoned instruction | What can each tool do, and what requires human confirmation? |
Deployment adds a second axis
- Provider-hosted inference puts training data, weights, and internal logs outside the reviewer's visibility, regardless of how the layer above is scoped.
- Self-hosted inference trades that gap for a direct one: patching, key management, and host access control become findings the organization must produce itself.
- Mixed workflows multiply boundaries rather than trading one for the other; each hop needs its own evidence.
- Providers routinely decline to name which model ran which step, citing trade secrets — plan to test behavior under adversarial input, not request a diagram.
Before scoping a review
- State explicitly which layer is in scope: model, service/application, or agent.
- For a model claim, ask about training data and known extraction risk.
- For a service claim, ask what is logged, retained, and shared with the provider.
- For an agent claim, map every tool's permissions and which actions require human confirmation.
- Identify cloud/local/hybrid boundaries and check each hop separately, not the deployment as a whole.
Frequently asked questions
If a vendor calls their product an 'agent,' does that settle the scope?
No. The label doesn't establish what's actually delegated. Ask what tools exist, what permissions each one has, and which actions run without a human checking first — that answers the scope question better than the product name does.
Does reviewing the service layer cover the model underneath it too?
Only partially. Service-layer review catches logging, retention, and API-boundary issues, but it won't surface what's memorized in the model's weights or how the model behaves when a different application wraps it.
Related
AI security
How to secure GenAI, LLM and agentic AI systems: current OWASP threat classes, layered controls, agent authority, retrieval isolation and monitoring.
Agentic threat models
Why the shift from generative query-response to an autonomous agentic loop changes the threat model of an AI system from bad output to unintended action.
AI risk map
Eight places where AI systems retain, transform or expose data—and the questions that reveal whether each one is controlled.
Deleting model data
Why trained data is hard to remove from LLM weights, and how incident responders should contain, replace and test an affected model.
AI security assessment
Security assessment of GenAI and LLM systems: prompt injection, tool and agent abuse, data leakage, model supply chain and monitoring gaps.
Threat modeling
Facilitated STRIDE threat modeling on your real architecture, plus the templates and coaching to keep teams doing it without me.