Reading what a probe actually sends

A probe's goal is a sentence. The prompts are the thing, and you can read them without opening any source code.

Point the probe at test.Repeat, the generator that answers with whatever it was given. The report then holds the prompts as if they were answers.

Example

garak -t test -n Repeat --spec probes.goodside.WhoIsRiley --generations 1 --report_prefix seen > /dev/null 2>&1
python3 -c "
import json
rows = [json.loads(l) for l in open('$HOME/.local/share/garak/garak_runs/seen.report.jsonl')]
sent = [r['prompt']['turns'][0]['content']['text'] for r in rows if r.get('entry_type') == 'attempt']
for text in sorted(set(sent)):
    print('-', text[:64])
"

Six prompts, all variations on asking about a person the probe knows the model should not claim to know. Reading them is the fastest way to decide whether a probe is relevant to your application.

Why this matters before a real run

A probe that sends 160 prompts against a paid model, five times each, is 800 calls. Reading the prompts first tells you whether those calls are worth making, and a probe about a quirk of one model family is worth skipping when you run another.

This also shows what leaves your machine. Some probes send text you would not want in a third party's logs. Reading the prompts before the first real scan is the same discipline as reading a query before running it on production.

Try it yourself

Do the same for probes.goodside.Tag and look at what makes it 160 prompts.
Count the distinct prompts against the number of attempts in the report.
Pick a probe you would not run against a customer's system, and say why.

PreviousWhat a probe declares about itself

Every expert started right here.