LLMOpsThe LLMOps lifecycle · MLflow 3 · FastAPI · Docker · AWS/Azure
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
41 small wins to finish your pathNext lesson →

What to evaluate: RAG vs a plain LLM

What you evaluate depends on the app: a RAG system is scored on whether its answer is grounded in what it retrieved, while a plain LLM is scored on whether its answer is correct and appropriate.

Last updated: 15 Oct, 2026 · LLMOps

The Logging a GenAI run to MLflow lesson recorded a faithfulness score without saying why that score and not another. The LLMOps lifecycle video says the evaluation step uses different metrics for a RAG application than for a simple LLM model, and that both sets can apply. This lesson names the two metric families so you pick the right ones before you log anything.

Why the two differ

A RAG app has two moving parts that can each go wrong: the retrieval step that fetches context, and the generation step that writes an answer from it. A plain LLM has only the generation step. So RAG needs metrics that judge the retrieved context as well as the answer, while a plain LLM is judged on the answer alone.

The RAG metric family

  • Faithfulness asks whether the answer is supported by the retrieved context, not invented. A low score is a hallucination on top of good or bad retrieval.
  • Context precision asks whether the retrieved chunks were relevant, rather than padding that crowds out the useful text.
  • Context recall asks whether retrieval found all the context the answer needed, so a correct question is not failed by a missing document.

The plain-LLM metric family

  • Correctness compares the answer against a reference or ground truth, the comparison made in the MLflow video.
  • Relevance asks whether the answer addresses the question, rather than being true but off-topic.
  • Toxicity and safety ask whether the answer contains harmful or disallowed content, a metric the MLflow video logs directly.
A RAG pipeline drawn left to right as question, retrieve, generate and answer, with a dashed arrow from each stage down to the metrics that judge it. Retrieve is judged by context precision and context recall, asking whether retrieval found the right chunks and all of them. Generate is judged by faithfulness, asking whether the answer is supported by what the retriever returned. The answer is judged by correctness, relevance and toxicity, asking whether it is right, on topic and safe to send. A panel underneath notes that a plain LLM has no retrieval step, so the first two families do not apply and it is judged on correctness, relevance and safety alone.

RAG metrics vs plain-LLM metrics

RAG appPlain LLM
Judges retrievalYes, precision and recallNo retrieval step
Judges groundingFaithfulness to the contextNot applicable
Judges the answerCorrectness and relevanceCorrectness and relevance
Judges safetyToxicityToxicity

Where you use each family

  • The RAG family for the shop assistant answering from a document store, where a wrong answer usually traces to bad retrieval.
  • The plain-LLM family for a summariser or classifier with no retrieval, where only the output is in question.
  • Both families when a RAG app also has to stay safe and on-topic, so you add the answer and safety metrics to the retrieval ones.
Watch out. A high answer score hides a retrieval problem. An answer can be correct because the model already knew the fact, while retrieval returned nothing useful; only the context metrics catch that, which is why a RAG app is never judged on the answer alone.
Try it yourself
  • List the shop assistant's failure modes and label each as a retrieval problem or a generation problem.
  • Read the metric list in Promptfoo overview and match each grader to the RAG or plain-LLM family.
  • Pick three metrics you would log for the shop assistant and write one sentence on what each would catch.
PreviousLogging a GenAI run to MLflow

You understood something today that you didn't yesterday.