One attempt, in full

The attempt is the unit of evidence. When someone asks what exactly the scanner sent and what came back, this is the answer.

Example

garak -t function -n support_bot#reply --spec probes.goodside.WhoIsRiley --generations 1 --report_prefix a > /dev/null 2>&1
python3 -c "
import json
for line in open('$HOME/.local/share/garak/garak_runs/a.report.jsonl'):
    row = json.loads(line)
    if row.get('entry_type') == 'attempt' and row.get('status') == 2:
        print('probe   :', row['probe_classname'])
        print('goal    :', row['goal'])
        print('prompt  :', row['prompt']['turns'][0]['content']['text'][:52])
        print('answer  :', row['outputs'][0]['text'][:52])
        print('scores  :', row['detector_results'])
        break
"

Five fields and the whole story: which probe, what it was trying to achieve, what it sent, what came back, and what each detector made of it.

status is worth knowing. An attempt is written twice, once when it is created and once when it is finished, and only the finished one carries answers. Filtering on status == 2 is why the loop above has that line in it.

The scores

detector_results is a dictionary of detector name to a list of numbers, one per answer. A probe with two detectors gives two entries, and the same answers are scored by both.

This is what you show a reviewer. A rate is an argument; an attempt is evidence. When a customer's security team asks what you actually tested, you open the report and read them the prompt and the answer.

Try it yourself

Print every attempt with a non-zero score and nothing else.
Compare the count of status 2 attempts with the total_evaluated in the eval line.
Find the uuid on an attempt and explain why it is there.

PreviousThe report file

Every expert started right here.