What a target must return

One function, two rules, and a handful of ways to get it wrong. This lesson is the contract, because every strange scan later starts here.

The signature

def reply(prompt, **kwargs):
    return ["one answer"]

The prompt arrives first and as a string. **kwargs catches everything else garak passes, and a function without it fails the moment garak sends a setting it did not expect.

The return

A list, and one entry per answer. Returning a bare string is the commonest mistake, and garak treats the string as the sequence it asked for, so every character becomes an answer.

cat > wrong_bot.py <<'PY'
def reply(prompt, **kwargs):
    return "a string, not a list"
PY
garak -t function -n wrong_bot#reply --spec probes.test.Blank --generations 1 --report_prefix wrong 2>&1 | tail -4

Twenty characters became twenty answers, and garak says so: it asked for one response and got twenty. The message names the count rather than the cause, so the first time you see it, look at what your function returns.

Returning nothing

A generator may return [None] when it has no answer, for a refusal or a filtered response. There is then nothing for a detector to score.

cat > none_bot.py <<'PY'
def reply(prompt, **kwargs):
    return [None]
PY
garak -t function -n none_bot#reply --spec probes.test.Blank --generations 1 --report_prefix none 2>&1 | tail -4

The verdict is SKIP, and the count is nothing out of nothing. A target that answers nothing is not a target that passed, and garak keeps the two apart rather than flattering the score.