Framing the AI attack surface
An LLM application is more than a model. It is a system of prompts, retrieval sources, tools and plugins, downstream integrations, and output consumers. Security testing has to consider each layer because impact usually comes from how model output is trusted and acted upon, not from the model in isolation.
Two public references anchor the work: the OWASP Top 10 for LLM Applications (https://genai.owasp.org) for common weaknesses, and MITRE ATLAS (https://atlas.mitre.org), an ATT&CK-style knowledge base of real-world adversarial machine learning tactics and techniques.
Prompt injection and insecure output handling
Prompt injection (LLM01) is the signature LLM risk: untrusted input causes the model to ignore its instructions or perform unintended actions. It appears as direct injection in user input and indirect injection through content the model retrieves, such as documents, web pages, or emails.
Insecure Output Handling is the paired risk. If model output is passed to a browser, shell, SQL query, or downstream API without validation, prompt injection can escalate into cross-site scripting, command injection, or SSRF. Test the sink, not just the model.
- Test direct injection that attempts to override system instructions.
- Test indirect injection by planting instructions in documents or pages the model will read.
- Trace model output into every downstream sink and check for missing encoding or validation.
Data and model risks
Sensitive Information Disclosure covers leakage of training data, system prompts, secrets, or other users' data through model responses. Training Data Poisoning targets the integrity of data used to train or fine-tune, and Supply Chain risks cover compromised models, datasets, and plugins pulled from third parties.
Testing here checks what the model can be induced to reveal, whether tenant isolation holds in retrieval-augmented systems, and the provenance and integrity controls around models and data sources.
- Attempt system prompt and configuration extraction.
- In RAG systems, test whether one tenant can retrieve another tenant's documents.
- Verify provenance, signing, and integrity checks for models, datasets, and plugins.
Agents, tools, and excessive agency
Excessive Agency (LLM08) is the risk that a model is granted too much autonomy, permission, or functionality, so that a manipulated model can take consequential actions. Insecure Plugin Design compounds this when tools accept free-form input or run with broad privileges.
For agentic systems, testing focuses on least privilege: what tools the agent can call, what those tools can do, and whether a human approval step gates high-impact actions such as sending money, deleting data, or executing code.
Governance alignment and reporting
AI security testing is strongest when tied to a governance framework. The NIST AI Risk Management Framework (https://www.nist.gov/itl/ai-risk-management-framework) and ISO/IEC 42001 provide the management-system context, while ATLAS supplies adversarial technique coverage for red-team style testing.
Findings are reported with the OWASP LLM category, the ATLAS technique where applicable, reproduction prompts and payloads, the downstream impact, and remediation that typically hardens input trust boundaries, output handling, and tool permissions.
Related
FAQ
What is the difference between direct and indirect prompt injection?
Direct prompt injection is malicious instruction placed in the user's own input to the model. Indirect prompt injection hides instructions in external content the model later reads, such as a web page, document, or email, so the attacker never talks to the model directly.
What is MITRE ATLAS?
MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a knowledge base of adversarial machine learning tactics and techniques modeled on MITRE ATT&CK. It helps teams describe and test how real attackers target AI systems.
Why is insecure output handling as important as the model itself?
Because impact usually comes from what happens to model output. If output flows unvalidated into a browser, shell, database query, or API call, a prompt injection can turn into XSS, command injection, or SSRF, so the downstream sink must be tested and hardened.