A real judge: llm_factory and an API key

Every score in this course came from a judge that compares words, so that none of it needed an account. Here is the swap, which is two lines.

from pretend_ragas import PretendEmbeddings, PretendJudge

judge = PretendJudge()
embeddings = PretendEmbeddings()

from openai import AsyncOpenAI
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory

client = AsyncOpenAI()
judge = llm_factory("gpt-4o-mini", client=client)
embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

llm_factory builds the judge and embedding_factory the embedding model. Every metric you wrote keeps its shape: the arguments llm= and embeddings= now hold real models.

export OPENAI_API_KEY="sk-..."

What changes

• Scores stop repeating exactly. The same row can score 0.8 then 0.75, so compare runs by direction and size, not by the last decimal.
• Judgement gets better. The shorter answer in lesson 8 stops being punished for missing a word, because a real judge reads meaning.
• It costs per row. Faithfulness is two calls per sample, relevancy three plus embeddings. A hundred rows is a few hundred calls.
• Other providers work the same way. llm_factory takes a provider argument for Anthropic, Google and anything OpenAI-compatible, including Ollama on your own machine.

Keep the stand-in in the repository. It is the fastest way to check that a new metric is wired up correctly before you spend anything on it.