When should you use classical ML instead of an LLM in a system?

30-second answerSay your answer out loud first, then reveal.

Use classical ML (gradient boosting, logistic regression, small fine-tuned encoders) when you have plenty of labelled data, structured or tabular features, very high volume, tight latency or cost constraints, or a need for calibrated, explainable, deterministic predictions, as in fraud scoring, demand forecasting, ranking and churn prediction. Use LLMs for unstructured language understanding and generation, few-data tasks, and fast prototyping. Many systems combine both.

Factor	Classical ML favoured	LLM favoured
Data type	Tabular, numeric, structured	Text, images, unstructured
Labelled data	Abundant	Scarce (zero/few-shot)
Volume / latency	Millions per second, < 10 ms	Lower volume, seconds OK
Cost per prediction	Must be tiny	Higher acceptable
Explainability / calibration	Required (credit, regulation)	Less critical
Task	Score, rank, forecast	Summarise, extract, converse, reason over text

Hybrid patterns

An LLM generates labels or features → train a cheap classifier (distillation).
Classical ML scores risk → an LLM explains the decision to analysts (Q40).
LLM embeddings as features into a gradient-boosted model.
A small classifier routes; an LLM handles only complex cases.

Interview signal. Not reaching for an LLM by reflex. "For fraud scoring at 50K TPS, I'd keep the XGBoost model; the LLM helps investigators review flagged cases."

PreviousHow should prompts be managed in production?

Slow is fine. Stopping is the only problem.

When should you use classical ML instead of an LLM in a system?

Hybrid patterns

Related