RAG vs fine-tuning vs long context: when do you use each?

30-second answerSay your answer out loud first, then reveal.

Use RAG for knowledge: facts that change, private data, large corpora, and anywhere citations matter. Use fine-tuning for behaviour: style, format, domain language, or a narrow task done cheaply with a smaller model. Fine-tuning is a poor way to add facts. Use long context when the relevant material is small enough to fit and you need the model to reason over all of it at once (one contract, one codebase module). They combine well.

Need	Best fit	Why
Answer from 50K internal docs that change weekly	RAG	Updatable, scalable, citable
Always respond in a specific JSON format / tone	Fine-tuning (or just prompting)	Behaviour, not knowledge
Medical terminology the model misunderstands	Fine-tune the embedding model, or the LLM	Domain language
Analyse one 200-page contract thoroughly	Long context	Fits; needs holistic reasoning
Cheaper, faster model for one narrow task	Fine-tune / distil a small model	Cost and latency
Access-controlled data	RAG	Filter per user at retrieval time

Why fine-tuning is poor for facts

Models learn new facts unreliably from fine-tuning and may still hallucinate them.
There are no citations, and you can't update or delete one fact easily (a GDPR problem).
It requires retraining for every data change.

Combined approaches

RAG + a fine-tuned generator that's better at using context and citing.
RAG + long context: retrieve whole relevant documents instead of tiny chunks.
Fine-tuned embedding models for domain retrieval (Q42).

Interview framing. "RAG is for what the model should know, fine-tuning is for how it should behave, long context is for what it should read right now."

PreviousHow do you prompt the LLM so it answers only from retrieved context and cites sources?

You understood something today that you didn't yesterday.

RAG vs fine-tuning vs long context: when do you use each?

Why fine-tuning is poor for facts

Combined approaches

Related