1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
Your POC reaches 70% accuracy, but the customer expected 95%. What do you do?
30-second answerSay your answer out loud first, then reveal.
Step 1 — Error analysis (example)
| Cause | Share of errors | Fixable? |
|---|---|---|
| Ground-truth labels inconsistent between SMEs | 25% | Yes: clarify the definitions (true accuracy may be higher) |
| Scanned documents poorly parsed | 30% | Partially: better OCR / vision model |
| Missing context (info not in documents) | 20% | No: needs a process change or data source |
| Model reasoning errors on complex cases | 25% | Partially: prompts, examples, stronger model |
Step 2 — Options to present
- Improve: 2 more weeks targeting the fixable categories, with a projected 82–88%.
- Confidence-based routing: the system handles 60% of cases at 96% accuracy automatically and routes 40% to humans, which still saves most of the manual effort.
- Narrow scope: typed documents only, where accuracy is already at 90%.
- Assist mode: AI pre-fills, humans verify everything. Time savings without autonomy risk.
Step 3 — Reset expectations honestly
Compare with human accuracy on the same task (often 85–95%, not 100%), and focus on the business metric (time saved, errors caught), not the raw accuracy number.
Interview signal: data-driven diagnosis, creative options, honest communication, and no "we'll just fine-tune it to 95%".
Related
- Previous: Q17. You have two weeks for a POC, and the customer's data is messy: scanned PDFs, Excel files with merged cells, and emails. What do you do?
- Next: Q19. The customer insists the AI must be fully autonomous, with no humans reviewing outputs. You think that's risky. How do you handle it?
- DeepEval
- RAG Interview Questions
Slow is fine. Stopping is the only problem.