Your own app as the target

The generators in lesson 3 answer nothing useful. The thing worth scanning is your application, and garak will call any Python function you point it at.

A generator is anything that takes a prompt and gives back text. The function type takes a module and a function name, separated by a hash.

python

HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def reply(prompt, **kwargs):
    asked = prompt.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return [line]
    return ["I can only help with orders and refunds."]

Two rules it must follow. It takes the prompt first and accepts **kwargs, because garak passes settings it does not need to know about. And it returns a list of strings, not a string, because one prompt can be asked several times.

Example

garak -t function -n support_bot#reply --spec probes.test.Blank --report_prefix scan

The verdict says the detector found something in all five answers. That is not a bug in the bot: test.Blank sends an empty prompt and the detector behind it reports any output at all, so a bot that says anything is a hit. It is a test of the harness, not of the assistant.

Read the generator name in the output. It says function:support_bot#reply, which is garak telling you what it actually loaded. When a scan behaves strangely, check that line first; a typo in the function name is a different error than a bad scan.

Why this is the generator that matters

Everything your application does before the model sees a prompt, and everything it does to the answer afterwards, is inside that function. A scan that talks to the model directly skips your system prompt, your retrieval, your output filtering and your formatting, and those are where most of the interesting failures live.

Several answers per prompt

Garak asks each prompt five times by default, because a model does not answer the same way twice. The bot here is deterministic so the five are identical, and --generations 1 makes runs faster while you are writing.

Example

garak -t function -n support_bot#reply --spec probes.test.Blank --generations 1 --report_prefix one

Five attempts became one, and the run is quicker. Put it back up when the answers actually vary.

Try it yourself

Return a string instead of a list and read the error.
Rename the function and watch the loading line fail.
Add a print inside the function to see exactly what garak sends.

PreviousInstall garak and scan something

This is what real progress feels like.