The report file

Every line the scan printed is a summary of a file it wrote. The file is the scan; the terminal is a courtesy.

Example

garak -t function -n support_bot#reply --spec probes.test.Blank --generations 1 --report_prefix r > /dev/null 2>&1
python3 -c "
import json, collections
rows = [json.loads(l) for l in open('$HOME/.local/share/garak/garak_runs/r.report.jsonl')]
print(collections.Counter(r.get('entry_type') for r in rows))
"

One JSON object per line, and the entry_type says what each is. Four of them matter.

entry_type	What it holds
init and start_run setup	the version, the arguments, the whole configuration
attempt	one prompt, its answers and the detector scores
eval	one probe and detector pair, with the counts
probe_summary	the same, aggregated per probe
digest	the scored summary the HTML report is built from

The counts

Example

garak -t function -n support_bot#reply --spec probes.test.Blank --generations 1 --report_prefix r > /dev/null 2>&1
python3 -c "
import json
for line in open('$HOME/.local/share/garak/garak_runs/r.report.jsonl'):
    row = json.loads(line)
    if row.get('entry_type') == 'eval':
        print({k: row[k] for k in ('probe', 'detector', 'passed', 'fails', 'total_evaluated')})
"

That line is the verdict from the terminal, in a form a program can read. passed and fails are over answers rather than prompts, and nones holds the answers that were nothing at all.

Keep the report, not the terminal. It is the only thing that lets you say this week is better than last, and it is what the CI job in lesson 20 reads. The HTML file beside it is generated from the same lines.

Try it yourself

Print the init entry and find the garak version in it.
Count the attempt lines and check the number against the verdict.
Open the .report.html in a browser and find the same numbers.

PreviousJudging the answers yourself

This is what real progress feels like.