Red-team report: attacking the support assistant before a customer does
A security review asks one question about an AI assistant: what happens when someone tries to break it? This project answers with numbers from real attacks, before and after a fix.
The brief
The support assistant is about to go live on a public website. The customer's security team will not approve it without a report showing it was attacked, what got through, and what was done about it.
What you will build
- A garak scan of the assistant for prompt injection, jailbreaks and data leakage
- Two PyRIT attacks written for this assistant, such as extracting another customer's order or getting a refund promised
- A fix for the worst finding, in guardrails or in the tools
- A second run showing the attack success rate after the fix
- A two-page report for a security reviewer
Requirements
| Requirement | Done when |
|---|---|
| Scan | garak runs against the assistant itself, not a bare model, and its report is kept |
| Targeted attacks | At least two PyRIT attacks specific to this assistant's tools and data |
| Scoring | Each attack's success is judged by a scorer, and the scorer was checked on a few examples by hand |
| Fix | One finding is fixed and the same attacks are run again |
| Report | States attack success rates before and after, what is still open, and the risk accepted |
What to learn it from
- garak: scanning for known attacks and reading attack success rate
- PyRIT: scripted attacks, scorers and converters
- Guardrails AI or NeMo Guardrails: the fix on inputs and outputs
- Agent Governance Toolkit: the fix when the risk is a tool call
Milestones
- Wrap the assistant so garak and PyRIT can call it
- Run the garak scan and record the baseline
- Write and run the two targeted attacks
- Pick the worst finding and fix it
- Run everything again with the same settings
- Write the report
Only attack what you own
Run these tools against systems you built or have written permission to test. Scanning someone else's assistant is an attack, not a test.
Stretch goals
- Fail the build when the scan's attack success rate rises
- Add an indirect prompt injection hidden in a document the assistant retrieves
- Measure how much the fix slowed down normal answers
Before you call it done
Worth remembering
- The same attacks were run before and after the fix
- Every number in the report came from a saved run
- The report says what is still open, not only what was fixed
Every expert started right here.