Swapping the stand-ins for the real thing

Every runnable piece of this course is a stand-in, chosen so the lessons need no key, no GPU and no account. Each one has a real counterpart that drops into the same seam: a provider call, a vector store, a judge metric and a database-backed tracking store.

Last updated: 15 Oct, 2026 · LLMOps

Maintenance and drift after you ship closed the loop on an app that is live. This lesson is the hand-over from the course's version of that app to yours. Nothing about the shape changes: the config, the prompt library, the run record and the gate all stay where they are, and four pieces behind them get replaced one at a time.

The four stand-ins, and what replaces each

The first three need a key, so they are named here rather than run. The fourth needs nothing, so it is the one you can swap right now.

The swap you can run: a database-backed store

Syntax: the tracking URI is the only line that changes. A SQLAlchemy URI puts the same runs in a database file instead of a folder tree, which is what the MLflow project recommends for anything that outlives a lesson, and every logging call stays identical.

import os
os.environ["MLFLOW_DISABLE_AGENT_HINT"] = "1"    # hush MLflow's agent hint while you learn
import shutil
shutil.rmtree("mlruns", ignore_errors=True)      # start from an empty store
if os.path.exists("mlflow.db"):
    os.remove("mlflow.db")                       # and from an empty database

import mlflow

# the one line that changes: a database URI in place of file:./mlruns
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("shop-assistant")

# every call below is the one you already wrote
with mlflow.start_run(run_name="on-sqlite"):
    mlflow.log_param("model", "shop-stand-in")
    mlflow.log_metric("answer_length", 8)

runs = mlflow.search_runs(experiment_names=["shop-assistant"])
print("backend :", mlflow.get_tracking_uri())
print("runs    :", len(runs))
print("db file :", os.path.exists("mlflow.db"))

backend : sqlite:///mlflow.db
runs    : 1
db file : True

What the swap showed

• Only the URI moved. start_run, log_param, log_metric and search_runs are the same calls against a different backend, which is what makes this a seam rather than a rewrite.
• The runs survive a folder delete, because they are rows in mlflow.db rather than files under mlruns/.
• It is the step before a server. A hosted backend, the subject of A remote tracking server, is the same change again with an HTTPS URI and a token.

Swapping the other three

• The model. call_api takes a prompt and returns a string, so a provider call replaces its body, and the call sites do not move. Routing it through a gateway buys retries, fallbacks and one place to see the spend: LiteLLM overview.
• Retrieval. The handbook dict stands where a retriever goes. Replacing it means the indexing path from The RAG data pipeline built for real, with the embedding model pinned as part of the index.
• The score. Word overlap is a stand-in for a judge. A real metric calls a model and returns a score with a reason: DeepEval overview and RAGAS overview both plug into the same gate.
• Swap one at a time. Each replacement changes what the numbers mean, so change one, re-run the evaluation set, and keep the run that proves the new piece did not make things worse.

Related

• Previous: Maintenance and drift after you ship
• See also: Project: the production assistant
• See also: A remote tracking server