1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
You need an eval set for a new domain (e.g. insurance claims Q&A) within one week, with no labelled data. How do you do it?
30-second answerSay your answer out loud first, then reveal.
One-week plan
| Day | Activity |
|---|---|
| 1 | SME interview (1–2 hrs): top 20 question types, critical errors, tricky cases; collect policy docs, sample emails or tickets |
| 2 | Draft taxonomy (intents × difficulty × policy type × language); extract real questions from historical data |
| 3 | LLM-generate additional questions per gap cell, grounded in specific policy sections; draft reference answers with citations |
| 4 | Automated filtering (dedupe, check citations exist); prepare review sheet |
| 5 | SME review session(s): approve, fix or discard; mark critical facts per answer |
| 6 | Define rubric and judges; calibrate on 30–40 items against SME grades |
| 7 | Baseline run; error analysis; document dataset v0 and its known gaps |
Quality tips
- Include unanswerable questions (not covered by the policy) and ambiguous ones (needing clarification).
- Mark critical facts in references (amounts, waiting periods, exclusions) so judges check them specifically.
- Keep generated questions realistic: add colloquial and Hinglish variants.
- Record provenance (real vs synthetic), and analyse the two separately.
Related
This is what real progress feels like.