AI security
AI security is application security where the untrusted input can rewrite the program. The useful mental model is authority: what the model can do, on whose behalf, and what confirms the action.
Updated
Threat classes that matter in production
- Indirect prompt injection through retrieved documents, web pages, tickets or emails the model reads — still the top-ranked risk in the OWASP Top 10 for LLM Applications (LLM01:2025), now explicitly covering multimodal payloads in images, audio and files.
- Excessive agency: agents holding broader credentials than the requesting user, and system-prompt leakage that exposes the tool schema an attacker needs to abuse it.
- Cross-tenant leakage in retrieval indexes, embeddings and caches — its own OWASP category since the 2025 revision (LLM08: Vector and Embedding Weaknesses).
- Insecure output handling — model text rendered into HTML, SQL, shell or a downstream API call.
- Sensitive data flowing to inference providers without redaction or a data-processing basis.
- Cost and availability abuse through unbounded token consumption, denial-of-wallet against metered APIs.
- In agentic deployments: goal hijack, tool misuse, memory poisoning and unauthenticated inter-agent messages — the categories OWASP's 2026 Top 10 for Agentic Applications now names directly.
1Untrusted inputs
- User prompt
- Retrieved documents
- Web / email content
- Tool responses
2Trust boundary
- Input normalisation
- PII redaction
- Data / instruction separation
3Model + tools
- System prompt
- Delegated user tokens
- Confirmation on writes
- Step and budget limits
4Outputs
- Contextual encoding
- Schema validation
- URL / image containment
- Audit log
Layered controls
| Layer | Control | Failure it prevents |
|---|---|---|
| Input | Normalisation, length limits, PII stripping | Injection payloads, data leakage to provider |
| Retrieval | Per-tenant scoping, source labelling | Cross-tenant disclosure, poisoned context |
| Model | Hardened system prompt, refusal policy | Instruction override, persona hijack, prompt leakage |
| Tools / agents | Least-privilege, short-lived scoped credentials, human confirmation for writes | Excessive agency, destructive actions, identity abuse |
| Output | Contextual encoding, schema validation | XSS, injection into downstream systems |
| Runtime | Rate limits, quotas, audit logging, behavioural baselines | Cost abuse, undetected misuse, rogue or drifting agents |
Two taxonomies do most of the heavy lifting when I scope a review: the OWASP Top 10 for LLM Applications for single-model and RAG risk, and the OWASP Top 10 for Agentic Applications for systems where agents plan, hold memory and call tools with delegated authority. NIST's Generative AI Profile (NIST AI 600-1) supplies a complementary risk catalogue — covering confabulation, information integrity and value-chain exposure — for teams running a formal AI RMF programme rather than a point-in-time assessment.
Frequently asked questions
Can prompt injection be fully solved?
No. Treat model output as untrusted and constrain what actions it can trigger; that turns injection from a breach into a nuisance.
Do guardrail products help?
They raise the cost of trivial attacks. They do not substitute for authorisation design.
Is agentic AI a different threat model?
Agents add delegated authority, memory and inter-agent messaging to the same underlying weaknesses, so identity, tool scoping and behavioural monitoring matter as much as prompt hygiene.
Related
AI security assessment
Security assessment of GenAI and LLM systems: prompt injection, tool and agent abuse, data leakage, model supply chain and monitoring gaps.
AI governance & AI Act readiness
Build a workable AI governance framework: system inventory, risk classification, acceptable-use policy, review gates and AI Act evidence.
Prompt injection defence
Why prompt injection cannot be filtered away, why it has no full fix yet, and how to design AI systems — architecturally, not just with better prompts — so a successful injection has no useful consequence.
LLM security controls
A control checklist for enterprise GenAI: data handling, retrieval isolation, agent authority, cascading-failure containment, monitoring, cost limits and vendor terms.
Threat modeling
A practical STRIDE-based threat modeling method for cloud, application and AI systems, built to be run by engineering teams.
STRIDE for AI systems
Applying STRIDE to LLM and agentic applications: what each category means when untrusted text can act as instructions, where STRIDE stops covering multi-agent failure modes, and the controls per element.