Release gate: an eval suite that blocks bad releases
The answer to "how do you know it works" is a number that is checked before every release.
The brief
The policy assistant from an earlier stage is about to be used by the whole company. The HR lead asks for proof that each update does not make answers worse.
What you will build
- A golden set of at least 30 real questions with expected behaviour
- Automated scoring for correctness, citations and refusals
- A check that fails the build when the score drops
- A one-page report a stakeholder can read
Requirements
| Requirement | Done when |
|---|---|
| Golden set | Built from real questions, including ones that must be refused |
| Scoring | Runs without a person, with the scoring method written down |
| Gate | A deliberately broken prompt fails the check |
| Report | Shows the score over time and the worst failures |
What to learn it from
- Topic: golden datasets
- DeepEval or Ragas: scoring answers with a judge
- Promptfoo: regression suites in CI
- OpenAI Agents SDK: tracing and streaming
Milestones
- Collect the questions and agree expected behaviour
- Write the scoring
- Run it once and record the baseline
- Add it to CI
- Break the prompt on purpose and watch it fail
- Write the report
How it is judged
Introduce a regression on purpose. If the gate lets it through, the project is not done.
Stretch goals
- Check the judge against human labels
- Track cost and latency alongside quality
- Alert when production answers drift
Before you call it done
Worth remembering
- The threshold was agreed, not invented
- A regression was caught for real
- A non-engineer understood the report
Every expert started right here.