Red teaming: what promptfoo redteam does

Everything so far has asked whether the bot answers your questions well. The other half of promptfoo asks what it does when somebody is not being nice, and it works differently enough to be worth a lesson even though this course does not run it.

An eval has questions you wrote. A red team has questions generated for you, from a description of what your application is supposed to do and a list of the things that could go wrong with it.

yaml

targets:
  - id: file://support_bot.py
    label: support-bot

The thing under test is called a target rather than a provider here, because a red team points at one application rather than comparing several. Below it goes the plan.

yaml

redteam:
  purpose: >
    A support assistant for an online shop. It answers questions about
    orders and refunds from a handbook, and nothing else.
  numTests: 5
  plugins:
    - pii
    - hallucination
    - excessive-agency
  strategies:
    - jailbreak

purpose is the important field. It is the description everything else is generated from, and a vague one produces vague attacks. plugins name the risks to probe for, strategies name the ways of dressing an attack up, and numTests is how many of each.

The two commands

bash

promptfoo redteam generate
promptfoo redteam run

generate writes the test cases to a file, which you can read before anything is sent anywhere. run generates and evaluates in one go. The results open in the same viewer as everything else, arranged by risk rather than by question.

Why it is not in this course

Generating attacks is a model's job, and it happens on promptfoo's servers or with your own key. There is no local stand-in for it, because the value of a generated attack is that you did not think of it. A course that faked them would be teaching the menu rather than the meal.

Point it only at systems you are allowed to test. That is your own application, in your own environment, with the people responsible for it aware that it is happening. Red teaming somebody else's service is not a testing exercise.

What the plugins cover

There are around seventy, grouped into families: leaking personal data, producing harmful content, doing things it was not meant to do, being talked out of its instructions, and the compliance sets that map to OWASP's LLM top ten, NIST's AI risk framework and MITRE ATLAS. You pick the families that match what your application can actually do, because a bot that cannot take an action does not need the tests about taking actions.

The thing worth taking from this lesson even if you never run it: the guardrail checks from part 3 and the rubrics from part 4 are how you turn a red team finding into a test that stays. A finding is one bad answer. An assertion is a promise it will not come back.

Try it yourself

Write a purpose for an application you work on, in three sentences.
Read the plugin list in the promptfoo docs and pick the five that match it.
Take one failure you have already seen in production and write it as a test case in tests.csv.

PreviousContext assertions: faithfulness and recall

Slow is fine. Stopping is the only problem.