1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
Leadership asks you to cut the hallucination rate of a customer-facing assistant in half within a quarter. How do you run the program?
30-second answerSay your answer out loud first, then reveal.
Program plan
| Phase | Weeks | Work |
|---|---|---|
| Define and baseline | 1–2 | Taxonomy (fabricated fact, wrong number, unsupported policy claim, wrong citation); 500-case eval set; groundedness judge calibrated on 200 human labels; production sampling |
| Root-cause analysis | 2–3 | Categorise 150 failures by cause |
| Fixes (iterative) | 3–10 | Ordered by impact |
| Validate and roll out | 10–12 | A/B test; monitor; report |
Typical root causes and fixes
| Cause | Fix |
|---|---|
| Retrieval missed the relevant doc | Hybrid search, reranking, better chunking |
| Outdated or conflicting docs | Freshness pipeline, versioning, dedupe |
| Model answers without evidence | Abstention policy + examples; "answer only from sources" |
| Numbers, dates, prices | Pull from tools/DB; render via templates, not generation |
| Over-long answers add speculative content | Concise answer format |
| No verification | Post-generation groundedness check → regenerate or abstain |
Watch the trade-off: more abstention lowers hallucination but can hurt helpfulness. Track the answer rate and user satisfaction alongside it.
Reporting: "Unsupported-claim rate went from 8.2% (±1.5) to 3.6% (±1.1); critical errors from 2.1% to 0.6%; answer rate went from 94% to 91%."
Related
Every expert started right here.