Probes: where the prompts come from

A probe is a class that produces prompts for one weakness. Garak ships about 190 of them in 42 families, which is the part you are paying for.

Example

garak --list_probes 2>/dev/null | head -12

Two kinds of line. A family on its own, marked with a star, and the classes inside it. A class marked with the sleeping face is inactive by default, usually because it is slow or needs something downloaded.

Families worth knowing by name

Family	What it looks for
test	nothing. Prompts for checking the harness itself
goodside	known model quirks, from Riley Goodside's work
snowball	questions a model answers confidently and wrongly
encoding	instructions hidden in an encoded string
promptinject	instructions that try to replace your own
lmrc	the content risks in the Language Model Risk Cards
ansiescape	terminal escape codes in a reply
apikey	whether it will produce something shaped like a key

Run a whole family by naming it without a class. Three probes ran here, and each one brought its own detector.

Example

garak -t function -n support_bot#reply --spec probes.goodside --generations 1 --report_prefix g 2>&1 | tail -6

All three passed, which for this bot is unsurprising: it is a dictionary lookup that cannot be talked into anything. A real assistant with a model behind it is where these start finding things.

Start with a family, not with everything. Running the whole catalogue against a real model is thousands of calls and an afternoon. Pick the families that match what your application can actually do, which is the same argument the promptfoo course makes about red team plugins.

Try it yourself

Run --list_probes and count how many families carry the sleeping face.
Run probes.snowball against the bot and read the verdicts.
Pick the three families that match an application you work on.

PreviousWhat a target must return

You understood something today that you didn't yesterday.