Swapping the detector

The probe brings a detector. When it is the wrong one for your application, --detectors replaces it.

Here is a bot with a real problem: it answers with a wholesale price, which a customer-facing assistant should never do.

python

def reply(prompt, **kwargs):
    return ["Our wholesale price is 40 pounds a unit."]

Example

garak -t function -n leaky_bot#reply --spec probes.test.Blank --generations 1 --report_prefix leak 2>&1 | grep -E 'test.Blank'

A hit, but for the wrong reason. any.AnyOutput flags the answer because there is an answer, not because of what it says. The verdict is right by accident, which is worse than being wrong.

Naming the detector yourself

Example

garak -t function -n leaky_bot#reply --spec probes.test.Blank --detectors mitigation.MitigationBypass --generations 1 --report_prefix leak2 2>&1 | grep -E 'queue of detectors|test.Blank'

Now the question being asked is whether the bot refused, and it did not. The queue line confirms which detector ran, which is worth checking whenever a verdict surprises you.

Several at once

--detectors takes a comma separated list and every one of them scores every answer, so one scan can ask several different questions of the same replies.

The detector decides what the number means. Two scans of the same bot with the same probe can report 0% and 100% and both be correct, because they were asking different questions. A rate without the detector beside it is not a fact.

Try it yourself

Run the leaky bot with both detectors at once and read the two verdict lines.
Run always.Pass against it and notice the problem disappears.
Work out which detector would catch a price appearing in any answer.

PreviousDetectors: deciding whether it failed

You understood something today that you didn't yesterday.