AI and LLM security assessment
AI features fail in ways classic application security reviews do not look for: untrusted text becoming instructions, agents with more authority than the user, retrieval indexes leaking data across tenants. This assessment models the system as an attacker would and produces controls that survive contact with production.
Updated
Assessment areas
- Prompt and instruction integrity: direct and indirect prompt injection, system-prompt extraction, jailbreak resistance and untrusted content arriving through retrieval or tool results.
- Tool and agent authority: what the model can invoke, under whose identity, with what confirmation step, and whether downstream systems re-authorise the requested action.
- Retrieval and data boundaries: tenant isolation in vector stores, document ingestion, metadata filters, PII handling, redaction before inference and deletion paths.
- Model and dependency supply chain: model provenance, third-party plugins, model gateways, endpoint configuration and the trust placed in external tools or data sources.
- Output handling: rendering untrusted model output into HTML, SQL, shell or workflow contexts, and the controls that stop output becoming an instruction to another system.
- Abuse and cost controls: rate limiting, quotas, key handling, logging, retention and alerting for abnormal tool use or extraction attempts.
1Untrusted inputs
- User prompt
- Retrieved documents
- Web / email content
- Tool responses
2Trust boundary
- Input normalisation
- PII redaction
- Data / instruction separation
3Model + tools
- System prompt
- Delegated user tokens
- Confirmation on writes
- Step and budget limits
4Outputs
- Contextual encoding
- Schema validation
- URL / image containment
- Audit log
Method
01Model the system
Data-flow diagram with trust boundaries around the model, retrieval layer, tools, identities and users.
02Enumerate threats
STRIDE per element, plus AI-specific classes from the OWASP GenAI LLM Top 10. I identify where an attacker can turn data into instructions or turn model output into action.
03Test hypotheses
Hands-on probing of the deployed or production-like system: injection payloads, retrieval boundary crossing, tool invocation, authorisation and output handling.
04Design controls
Layered mitigations at input, model, retrieval, tool and output stages, expressed as engineering changes with acceptance tests rather than a generic prompt filter.
Frequently asked questions
Do you test the model or the system?
The system. Model-level robustness matters, but nearly every serious finding in production AI comes from the surrounding architecture: tool authority, retrieval scope and output handling.
Does this cover the EU AI Act?
Technically yes for security controls; the governance and classification side is covered by the AI governance and AI Act readiness engagement.
Related
AI governance & AI Act readiness
Build a workable AI governance framework: system inventory, risk classification, acceptable-use policy, review gates and AI Act evidence.
AI security
How to secure GenAI, LLM and agentic AI systems: current OWASP threat classes, layered controls, agent authority, retrieval isolation and monitoring.
Prompt injection defence
Why prompt injection cannot be filtered away, why it has no full fix yet, and how to design AI systems — architecturally, not just with better prompts — so a successful injection has no useful consequence.
STRIDE for AI systems
Applying STRIDE to LLM and agentic applications: what each category means when untrusted text can act as instructions, where STRIDE stops covering multi-agent failure modes, and the controls per element.
LLM security controls
A control checklist for enterprise GenAI: data handling, retrieval isolation, agent authority, cascading-failure containment, monitoring, cost limits and vendor terms.
Containing compromised agents
An agent will eventually be tricked, mistaken, or hijacked. Five architectural controls decide what it can still do when that happens.