How do you evaluate LLMs used for classification tasks (intent detection, routing, tagging)?

Example confusion insight

Billing ↔ Refund confusion suggests overlapping definitions. Fix the label descriptions and examples, or merge the classes if downstream handling is similar.

Metrics to know

• Precision: of items predicted X, how many are X (cost of false alarms).
• Recall: of actual X, how many were found (cost of misses).
• Macro vs micro averages: macro treats classes equally (important for rare but critical classes such as "fraud" or "complaint").
• Calibration: does "0.9 confidence" mean right 90% of the time? LLM self-reported confidence is often poorly calibrated, so use logprobs or empirical calibration.

LLM-specific checks: output always in the allowed label set (constrained decoding or validation); stability across prompt paraphrases; the cost and latency vs a fine-tuned small classifier.

Related

• Previous: Q12. How can you detect hallucinations in LLM outputs?
• Next: Q14. What tools exist for LLM evaluation and guardrails, and how do you choose?
• Machine Learning
• DeepEval