How do you evaluate quality for multilingual and Indic-language users?

30-second answerSay your answer out loud first, then reveal.

Don't assume English results transfer. Build eval slices per language and script (Hindi in Devanagari, romanised Hinglish, Tamil, Bengali and others relevant to your users) from real user inputs, with native-speaker reviewers for references and grading. Measure task quality, language correctness (responding in the right language and script, fluency, appropriate formality), retrieval quality for cross-lingual queries, and guardrail performance (toxicity and PII detection often degrade in other languages). Validate any LLM judge per language.

What to measure per language slice

Dimension	Example check
Task accuracy	Same rubric as English, graded by native speakers or calibrated judges
Language match	Reply in the user's language and script (or as configured)
Fluency and naturalness	Native-speaker rating; avoid literal translations
Terminology	Domain terms (banking, legal) correct in the local language, or kept in English where users prefer
Cross-lingual retrieval	Hindi query → English doc retrieval recall
Guardrails	Toxicity / PII detection recall in that language; injection in other scripts
Cost / latency	Token inflation for non-Latin scripts

Practical tips

Code-mixed inputs ("mera refund kab aayega?") are extremely common in India; make them a separate slice.
LLM judges can be lenient or wrong in low-resource languages, so calibrate per language against native speakers.
Keep slice sizes large enough for meaningful comparisons (Q19).

PreviousYou must choose between two LLMs for a production feature. Design the evaluation and write the decision summary.

Little by little, you're building something great.

How do you evaluate quality for multilingual and Indic-language users?

What to measure per language slice

Practical tips

Related