A technical map of where AI risk concentrates
An AI deployment is not one asset. It is a chain of stores and processing stages, each with a different failure mode and a different containment job.
By Piotr Zenon Żabrowski · Published · Updated
The eight risk locations
| Location | What can go wrong | Primary control |
|---|---|---|
| Training corpus | Poisoned or sensitive material shapes later behaviour; rare sequences may be reproduced. | Provenance, quality gates and restricted corpus access. |
| Model weights | Memorised material is distributed across parameters, making targeted removal and incident recovery difficult. | Model provenance, extraction testing, version control and replacement plans. |
| Prompts and input files | Secrets enter logs or history; hostile instructions compete with trusted instructions. | Classification, redaction, retention limits and instruction/data separation. |
| RAG database | Over-broad retrieval leaks documents; poisoned chunks steer answers; embeddings expose information about source text. | Query-time authorisation, ingestion provenance, tenant isolation and retrieval tests. |
| Model output | Confident errors, reproduced secrets or unsafe code reach downstream systems. | Grounding checks, sink-aware encoding and human review for consequential use. |
| Service-side cache | Prompt representations, responses or agent state persist unexpectedly or cross a tenant boundary. | Cache isolation, expiry, encryption and documented invalidation behaviour. |
| Audit logs | Detailed prompts, tool parameters and outputs create a second high-value dataset. | Minimise fields, restrict access, define retention and test deletion. |
| Fine-tuning data | Bad examples or hidden secrets become durable behaviour in weights or adapters. | Dataset review, lineage, isolated training and rollback to a known model version. |
Due diligence that exposes the gaps
Ask the vendor or deployment team
- Which model and version run each task, where are they hosted, and which subcontractors receive data?
- What enters the system prompt, user prompt, conversation history and provider-controlled instruction layer?
- How is the context window assembled, truncated and protected from untrusted instructions?
- Does retrieval enforce the requesting user's permissions at query time, not only during ingestion?
- Where are prompts, files, embeddings, outputs, tool results and agent state stored, and for how long?
- Are any customer inputs used for training, fine-tuning, evaluation or human review?
- Which tools can read, write, send, delete or execute, and which effects require confirmation?
- Can the system loop across multiple steps, retain memory or transfer data between services?
- How are extraction, retrieval poisoning, prompt injection, cross-tenant access and unsafe actions tested?
- What happens if the model provider fails, changes behaviour or must be replaced?
The practical pattern is defence in depth: reduce exposed interfaces, isolate stores, constrain every action and keep an exit path from any single AI or ICT supplier. A current architecture diagram and evidence matter more than a generic security statement.
Frequently asked questions
Which location deserves review first?
Start where external input meets authority: prompts, retrieval and tool calls. Then trace every copy into caches, logs, memory and training pipelines.
Does self-hosting remove these risks?
No. It changes who operates the model boundary, but the corpus, weights, retrieval store, caches, logs and application permissions still require separate controls.
Related
AI security
How to secure GenAI, LLM and agentic AI systems: current OWASP threat classes, layered controls, agent authority, retrieval isolation and monitoring.
AI security assessment
Security assessment of GenAI and LLM systems: prompt injection, tool and agent abuse, data leakage, model supply chain and monitoring gaps.
Three layers, three risk profiles
A security review that skips naming its layer — model, service, or agent — ends up testing the wrong attack surface.
OSINT
Using open-source intelligence defensively: attack-surface discovery, executive exposure including AI-assisted pretexting, and awareness training that changes behaviour.
STRIDE for AI systems
Applying STRIDE to LLM and agentic applications: what each category means when untrusted text can act as instructions, where STRIDE stops covering multi-agent failure modes, and the controls per element.
Prompt injection defence
Why prompt injection cannot be filtered away, why it has no full fix yet, and how to design AI systems — architecturally, not just with better prompts — so a successful injection has no useful consequence.