Attack success rate

Every verdict ends in a number. It is the most quoted thing garak produces and the easiest to misread.

Example

garak -t function -n support_bot#reply --spec probes.test.Blank,probes.goodside.WhoIsRiley --generations 1 --report_prefix rate 2>&1 | grep -E 'ok on'

Two probes, two rates, and they are rates of different things. The first says every answer was a hit; the second says none were. Neither is a statement about the bot in general.

What the number is

The share of answers a detector scored above the threshold, for one probe and one detector. It is not a share of prompts, it is not across probes, and it says nothing about the probes you did not run.

So a scan reporting 0% means the probes you chose found nothing with the detectors they brought. That is a useful sentence and a much smaller claim than the assistant is safe.

Making it comparable

Fix the probe list. A rate from a different set of probes is a different measurement.
Fix the generations count. More answers per prompt is a different denominator.
Record the version. Probes change between releases, and so do the numbers.
Keep the report. The rate is a summary of it, and only the report survives a disagreement.

Do not put a single rate in front of a customer. Give it with the probes, the detectors and the date, or it will be read as a score out of a hundred for the whole system. That is the one misreading that matters.

Try it yourself

Run the same two probes with --generations 5 and see which number moves.
Run a family and count how many separate rates it prints.
Write the one sentence you would put under the number in a report.

PreviousOne attempt, in full

Little by little, you're building something great.