LLMOpsThe LLMOps lifecycle · MLflow 3 · FastAPI · Docker · AWS/Azure
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
41 small wins to finish your pathNext lesson →

Evaluation in the lifecycle

Evaluation in the lifecycle is not a one-time step: the same scoring runs as a gate before you ship, on a slice of live traffic as a canary, and continuously once the app is in production.

Last updated: 15 Oct, 2026 · LLMOps

The What to evaluate: RAG vs a plain LLM lesson chose the metrics. This lesson places them in time. The LLMOps lifecycle video puts evaluation and testing right before prompt engineering and deployment, and lists performance metrics, A/B testing and bias detection as part of it; the score it produces is what decides whether the app moves forward.

Three places evaluation runs

  • A pre-ship gate. Before a change is deployed, the evaluation set runs and the change ships only if the scores hold. This is where a regression is caught cheaply, before any user sees it.
  • A canary. The new version serves a small slice of real traffic while its scores are watched, so a problem the fixed evaluation set missed shows up on a few users rather than everyone.
  • Production monitoring. Scores keep being computed on live answers, because a model and its inputs drift, and an app that passed last month can fail today.
Three panels left to right, from before release to after it. The pre-ship gate runs on a fixed evaluation set, catches a regression and blocks the release. The canary runs on a slice of live traffic, catches what the set missed and rolls back. Production monitoring runs on all live traffic, catches drift over time, and alerts rather than blocking. A closing warning says a passing gate is necessary but not sufficient, because the set only covers the cases you thought of.

A/B testing and bias detection

Two of the checks the video names sit across these placements. A/B testing runs two versions against the same traffic or the same evaluation set and compares their scores, which is how you prove a prompt change helped rather than assuming it. Bias detection checks whether answers differ unfairly across groups, a safety metric you run before shipping and keep running in production.

Offline evaluation vs production monitoring

Pre-ship gateCanaryProduction monitoring
Runs onA fixed evaluation setA slice of live trafficAll live traffic
CatchesA regressionWhat the set missedDrift over time
Blocks a releaseYesRolls backAlerts, does not block
Cost if it failsNone, nothing shippedA few usersCaught early in the open

Where you use evaluation in the loop

  • A continuous integration check, running the evaluation set on every change so a drop in score fails the build.
  • A staged rollout, where the canary's scores decide whether the new version takes more traffic or rolls back.
  • A monitoring dashboard, tracking production scores so drift is seen as it starts, not after complaints.
Watch out. A passing gate is necessary, not sufficient. A fixed evaluation set only covers the cases you thought of, so an app that scores well offline can still fail on live inputs it never saw. Keep scoring in production rather than trusting the gate alone.
Try it yourself
  • Sketch where the shop assistant's evaluation set would run in a continuous integration pipeline.
  • Write one A/B question for the shop assistant, naming the two versions and the metric that would decide.
  • List one input the fixed evaluation set would miss that production monitoring would catch.
PreviousWhat to evaluate: RAG vs a plain LLM

Slow is fine. Stopping is the only problem.