1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
Design a CI/CD pipeline for an LLM application with evaluation gates.
30-second answerSay your answer out loud first, then reveal.
Gate examples
| Gate | Threshold |
|---|---|
| Golden-set quality | No significant drop vs baseline; ≥ target |
| Safety suite | 0 new critical failures |
| Format adherence | ≥ 99.5% valid JSON |
| Cost | Tokens per task ≤ baseline + 10% |
| Latency | p95 within SLO in staging load test |
Design notes
- Prompt and config changes go through the same pipeline as code, even if they're edited by non-engineers in a UI (the UI opens a PR or a versioned change request).
- Cost of evals: cache model outputs for unchanged components; run full suites on merge or nightly.
- Flaky statistics: use paired comparisons and confidence intervals, and require a manual override with justification when results are borderline.
- Data and index changes (new embeddings, re-chunking) also trigger retrieval evals.
Related
Little by little, you're building something great.