From a script to a harness

Did it work?

Changing the ask

What the run left behind

Learn PyRIT from scratch: targets, attacks, scorers, converters, the memory database, seeds and adaptive multi-turn attacks, ending in a leak-regression suite for an assistant you own. Every lesson runs with no API key.

PyRIT

Where the course goes

PyRIT tutorial: red team your own AI assistant, no API key

What you are going to build

The third probe found nothing because it was not really a third probe

Probe an AI assistant with plain Python, before PyRIT

Probing an assistant by hand

What is missing

Why a PyRIT harness beats a script of probes

What the script cannot do

What the objectives in this course look like

Install PyRIT and the rule about permission

Install PyRIT, and test only what you own

PyRIT memory: initialize_pyrit_async and the run database

The memory every run writes to

PyRIT PromptTarget: write a custom target with no API key

The assistant under test, as a PromptTarget

PromptSendingAttack: PyRIT's simplest attack

Your first attack

PyRIT output_attack_async: read an attack result

Reading a run

PyRIT scorer: turn an attack result into a verdict

Without a scorer the outcome is undetermined

Scoring something without running an attack

PyRIT Score object: value, rationale and category

What a score carries

PyRIT SelfAskTrueFalseScorer with no API key

A scorer that asks a model

Write a custom PyRIT scorer: MessageTrueFalseScorer

Writing a scorer of your own

PyRIT composite scorers: AND, OR and inverting

Combining scorers into one verdict

PyRIT converters: change a prompt before it is sent

A converter, on its own

PyRIT AttackConverterConfig: converters in an attack

A converter inside an attack

PyRIT converter families and which one to pick

The converter families, and picking one

Write a custom PyRIT converter for your own system

Writing a converter of your own

PyRIT memory: every prompt and reply is recorded

Everything is written to memory

Finding a run you no longer have the id for

Query PyRIT memory: get_message_pieces and filters

Querying what was sent and what came back

PyRIT memory: stored scores and attack results

Scores and attack results

PyRIT memory labels and the SQLite run database

Labels, and a database that outlives the process

Did it work? quiz

You set <code>SubStringScorer(substring="cannot share")</code> and your attack starts reporting <code>SUCCESS</code> against an assistant that is refusing correctly. What happened?