How do you run a red-teaming program for an LLM application?

30-second answerSay your answer out loud first, then reveal.

Define the threat model and harm categories for your product, then combine manual red-teaming (domain experts and security testers creatively attacking the system) with automated attack generation (LLM-generated adversarial prompts, known jailbreak libraries, fuzzing, injection payloads in documents and tools). Log successful attacks with severity, fix them through layered mitigations, convert them into regression tests in CI, and repeat before each major release and continuously after launch.

A red-teaming program flows from a threat model to a per-category test plan, through manual and automated attacks to findings, mitigations and a CI regression suite, with production monitoring looping back into the test plan.

Attack categories to cover: direct jailbreaks, indirect injection via documents, emails and web pages, system prompt extraction, data exfiltration (other users' data, secrets), unsafe tool actions, harmful content in your domain, bias and discrimination, resource abuse (very long inputs, loops), multilingual and encoded attacks (base64, leetspeak, Hinglish).

Good practice

Severity rubric (e.g. critical: data leakage or unauthorised action; low: mildly off-brand output).
Diverse testers: languages, cultures, domain experts.
Measure the attack success rate over time; fixes should reduce it without causing over-refusal.
Tools: Promptfoo red-teaming, PyRIT, garak, custom generators.

PreviousHow do you apply output guardrails when responses are streamed token by token?

Slow is fine. Stopping is the only problem.

How do you run a red-teaming program for an LLM application?

Good practice

Related