RAGASragas 0.4.3 · Python 3.10+
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
28 small wins to finish your pathNext lesson →

LLM evaluation

LLM evaluation is the systematic process of measuring how well an AI system performs against predefined objectives and expected outcomes, so you can judge its quality, effectiveness and reliability with numbers.

Last updated: 29 Sep, 2026 · RAGAS 0.4.3

A demo that answers three questions well tells you little about the thousand questions real users will ask. Evaluation replaces "it looks fine" with a score per question, per quality.

What evaluation is, and its two kinds · from the Production RAG Live Marathon · 173:57 to 179:50

Measuring against expected outcomes

The video reads the definition slowly. Measuring and assessing mean putting numbers on how well the app performs. Predefined objectives and expected outcomes are the goldens: the questions the app must handle, each with the answer you expect. Together they tell you where the app works and where it does not.

Evaluating the LLM vs evaluating the application

An AI application has two parts: the LLM that powers it, and all the code around the LLM. The video separates the evaluation of each.

  • Evaluating the LLM. Correctness, reasoning, hallucination, safety, bias and coherence of the model itself. This is mostly done for you: public benchmarks such as the leaderboards on artificialanalysis.ai and Hugging Face compare models on intelligence, speed and cost per task. You read them to choose a model.
  • Evaluating the application. Whether your whole pipeline, retrieval, prompt and model together, answers your users' questions the way you expect. No benchmark knows your catalog or your policies, so you build this yourself. The video calls this the part where most of your time goes, and it is what RAGAS is for.
LLM evaluationApplication evaluation
What is testedThe model on its ownYour retrieval, prompt and model together
Test dataPublic benchmark setsYour own goldens
Who runs itModel providers and leaderboardsYou
Question it answersWhich model should I use?Does my app answer my users correctly?

Why a RAG app needs application evaluation

A strong model can still give a wrong answer inside your app. The search can bring back the wrong chunk, the prompt can let the model add details the chunk never said, or the answer can drift off the question. A benchmark score for the model cannot see any of that; a score on your own questions can.

When to evaluate your app

  • Before the first release, to know the baseline quality.
  • After a change to the retriever, the chunking, the prompt or the model, to see whether the change helped.
  • When users report a bad answer: add that question to the goldens so it is checked from then on.
Watch out. Do not treat a model's benchmark rank as a score for your app. The same model can top a leaderboard and still answer your return-policy question wrongly because the search returned the shipping policy.
Try it yourself
  • Open artificialanalysis.ai/models and find where gpt-oss-20b, the course's judge, sits on intelligence and speed.
  • Write down three questions a TechNest customer might ask that a public benchmark would never contain.
PreviousInstallation and setup

You understood something today that you didn't yesterday.