Red Teaming AI Models

Red teaming is adversarial testing done before attackers do it for you. For an LLM application it means deliberately trying to make the system misbehave: leak data it should not, follow instructions it should not, or hand tools to people who should not have them. This guide covers attack categories, a repeatable testing process, the OWASP risk list, and how to document findings. It describes what the attacks look like at a category level and how to defend against them. It does not provide working exploit techniques.

Attack categories

Prompt injection is the core threat. The application shows the model trusted instructions, then the user or a retrieved document supplies untrusted text that tries to override those instructions. Direct injection comes from the user typing adversarial input. Indirect injection is worse: the malicious instruction hides inside a webpage, email, or document that your retrieval system feeds the model. Treat every retrieved document as untrusted input.

Jailbreaks are attempts to get the model to ignore its safety training. They matter most when your application layers its own guardrails on a general model. Data exfiltration covers the model revealing what it should not: system prompts, conversation history from other users, RAG chunks the current user has no permission to see, or memorized training data. Other categories include excessive agency (the model given tools and permissions it does not need), denial of service through expensive requests, and model theft via extraction queries.

A structured testing process

Start by scoping. Write down what the system does, what data it can reach, what tools the model can call, and who the users are. A customer support bot with no tools needs different tests than an agent with shell access. Define what counts as a failure: data leaving a trust boundary, an unauthorized action, output that violates policy.

Build test cases in layers. First, benign baseline: does the system behave correctly on normal input. Second, known attack patterns adapted to your domain, drawn from published threat taxonomies rather than improvised. Third, application specific cases: the weird trust boundaries your product actually has, like a multi-tenant RAG system where one tenant's documents must stay invisible to others.

Run the tests against a staging deployment with production-like data controls. Record every attempt and result. Then fix, and re-test to confirm the fix works and did not break normal behavior. Red teaming is a loop, not a milestone. Re-run it before every release that changes prompts, models, tools, or data access.

The OWASP Top 10 for LLM Applications

The OWASP GenAI Security Project publishes the Top 10 for LLM Applications, currently the 2026 edition released in August 2026. It ranks risks by community vote weighed against real incident data. Use it as the checklist your test plan maps to:

1. LLM01 Prompt Injection
2. LLM02 Sensitive Information Disclosure
3. LLM03 Excessive Agency
4. LLM04 Supply Chain (compromised models, packages, plugins)
5. LLM05 Data and Model Poisoning
6. LLM06 Unbounded Consumption (resource and cost exhaustion)
7. LLM07 Misinformation
8. LLM08 Hidden Context Exposure (system prompts and unseen context leaking)
9. LLM09 Vector and Embedding Weaknesses (retrieval manipulation)
10. LLM10 Improper Output Handling (executing or rendering model output without validation)

The ordering has shifted over editions as real incidents accumulate. Excessive Agency and Unbounded Consumption climbed as agents moved into production, which tells you where teams are getting hurt. Read the full document on the OWASP GenAI site for prevention and mitigation strategies per entry.

Defenses that actually hold

Some mitigations survive contact with reality. Treat model output as untrusted data: validate, sanitize, and never execute generated code or SQL without checks. Apply least privilege to tools: an agent that only reads files should not have write access. Put per-user rate limits and cost caps in front of inference so one user cannot exhaust the budget (this is LLM06). Separate trusted system instructions from untrusted user and retrieved content structurally, not just with polite wording in the prompt. For RAG systems, enforce access control at retrieval time so the model never sees documents the user cannot see.

What does not hold on its own: asking the model nicely in the system prompt to ignore bad instructions, and input filters that attackers iterate past in an afternoon. Prompt level instructions are a layer, not a boundary. Your real boundaries are code, permissions, and data flow controls.

Documenting findings

A finding worth fixing has five parts. The title and severity, using a consistent scale. The OWASP category it maps to, so patterns show up across reports. The reproduction steps, detailed enough that a developer can trigger it on the staging build. The impact in concrete terms: what data or action the attack reached. And the recommended fix with a code-level suggestion, not just "add validation."

Track findings like vulnerabilities, with owners and deadlines. Severity should weigh real impact over cleverness. An injection that reveals the system prompt of a public demo is interesting. An injection that exfiltrates another tenant's RAG documents is critical. After fixes ship, re-run the full test suite and confirm each finding is closed with evidence, not with a claim that the code looks safer now.

Running this as a team

Internal red teaming works if the testers are not the developers who built the system. Fresh eyes find the trust assumptions the builders stopped noticing. Schedule it as a regular exercise, keep a living test case library that grows with each round, and feed production incidents back into the test set. The goal is not a clean report. The goal is knowing, before your users find out, which failures your system is capable of.