Project: the production assistant

The production assistant is the course's shop support bot with the tracked path wired end to end: config and a prompt library feeding a keyless stand-in model, each answer scored in plain Python, and the run logged to MLflow with a gate that fails a bad answer. The logger, the custom exception and the FastAPI endpoint from Part 4 are left out here so the file stays readable in one screen; wiring them in is the last step of your own build.

Last updated: 15 Oct, 2026 · LLMOps

The Observability in production lesson was the last stage. This lesson collects the parts. The overview promised one example that grows across the course: a shop support assistant that begins as a plan, becomes a project with config, logging, typed errors, a prompt library and an API, then gets experiment tracking and evaluation with MLflow. Here that promise is a single runnable thing you can operate: structured, tracked and evaluated, on a keyless stand-in with no API key and no GPU.

The capstone changes config.yaml to name the keyless stand-in it runs on; the structuring part used a realistic model name, but the keys are the same and only the values differ.

The assembled assistant

One module holds the pieces from the structuring part: the config loader, the named prompt template, the knowledge base and the deterministic call_api stand-in, and a retrieval step that returns the handbook fact a question matches. On top of them sits evaluate_and_log, which answers a question, scores the answer for faithfulness, decides a pass or fail against a threshold, and logs the whole run to MLflow. This is the complete program the rest of the lesson runs.

import yaml, mlflow

def load_config(path="config.yaml"):
    with open(path) as f:                       # settings live in a file, not in code
        return yaml.safe_load(f)

# one named prompt template, filled with this turn's values
PROMPTS = {"support_answer":
    "You are a shop support assistant. Answer only from the facts.\n\n"
    "Facts:\n{context}\n\nCustomer question: {question}"}

def render(name, **values):
    return PROMPTS[name].format(**values)

# the knowledge base and a deterministic, keyless stand-in for the model
HANDBOOK = {
    "refund": "Refunds are issued within 30 days of delivery.",
    "shipping": "Standard shipping takes three to five working days.",
}

def retrieve(question):
    for keyword, fact in HANDBOOK.items():       # the retrieval step
        if keyword in question.lower():
            return fact
    return ""                                     # nothing retrieved: outside the handbook

def call_api(prompt):
    for fact in HANDBOOK.values():               # answer only from facts in the prompt
        if fact in prompt:
            return fact
    return "Please contact the shop for help with that."

def faithfulness(reply, context):
    # the share of the reply's words grounded in the retrieved context
    words = reply.lower().replace(".", "").split()
    grounded = sum(1 for w in words if w in context.lower().replace(".", "").split())
    return round(grounded / len(words), 2)

def evaluate_and_log(question, run_name, cfg):
    # assemble the pieces, score the answer, and log the run to MLflow
    context = retrieve(question)
    reply = call_api(render("support_answer", context=context, question=question))
    score = faithfulness(reply, context)
    verdict = "pass" if score >= 0.75 else "fail"   # the gate: a low score fails

    with mlflow.start_run(run_name=run_name):
        mlflow.log_param("model", cfg["assistant"]["model_name"])
        mlflow.log_param("prompt", question)
        mlflow.log_metric("faithfulness", score)
        mlflow.set_tag("verdict", verdict)
        mlflow.log_text(reply, "answer.txt")
    return reply, score, verdict

The settings live in a file the loader reads, so the model name is config, not a literal in the code.

assistant:
  model_name: "shop-stand-in"
  temperature: 0.0
retrieval:
  index_path: "indexes/shop-faq"
  top_k: 4

Pointing MLflow at a local store

The tracking setup is the local file store from the tracking part, under a shop-assistant experiment, wiped at the start so the record is the same every run.

import os
os.environ["MLFLOW_DISABLE_AGENT_HINT"] = "1"    # hush MLflow's agent hint while you learn
os.environ["MLFLOW_ALLOW_FILE_STORE"] = "true"   # 3.17 needs this opt-in for a local ./mlruns store
import shutil
shutil.rmtree("mlruns", ignore_errors=True)      # start every run from an empty store
os.makedirs("mlruns/.trash", exist_ok=True)

import mlflow
mlflow.set_tracking_uri("file:./mlruns")
mlflow.set_experiment("shop-assistant")

Answering a question the handbook covers

A question about refunds matches the handbook, so retrieval finds the fact, the stand-in answers from it, and the answer is grounded in its context. The run is logged and the gate passes.

from assistant import evaluate_and_log, load_config

cfg = load_config()

# a question the handbook covers: retrieval finds the fact, the answer is grounded
reply, score, verdict = evaluate_and_log("How do I get a refund?", "refund", cfg)
print("reply       :", reply)
print("faithfulness:", score)
print("verdict     :", verdict)

reply       : Refunds are issued within 30 days of delivery.
faithfulness: 1.0
verdict     : pass

The failure mode: a question the handbook misses

A question the handbook does not cover is where the assistant is wrong, and the point is that the gate sees it. Retrieval returns nothing, so the stand-in falls back to its generic reply, which is not grounded in any context. Faithfulness drops to zero and the gate records a fail.

# a question the handbook does not cover: retrieval returns nothing, so the
# stand-in falls back and the answer is not grounded in any context
reply, score, verdict = evaluate_and_log("Do you price match other shops?", "price-match", cfg)
print("reply       :", reply)
print("faithfulness:", score)
print("verdict     :", verdict)

reply       : Please contact the shop for help with that.
faithfulness: 0.0
verdict     : fail

Reading the tracked record back

With both runs logged, the record reads back with search_runs: the two prompts, their faithfulness and the gate's verdict side by side, so the bad answer is caught in the tracking, not only in the moment it was given.

# read the tracked record back: the gate's verdict is logged, not guessed
runs = mlflow.search_runs(experiment_names=["shop-assistant"])
print("runs recorded:", len(runs))
for _, r in runs.sort_values("metrics.faithfulness").iterrows():
    print(f'  {r["tags.verdict"]}  faithfulness={r["metrics.faithfulness"]}  prompt={r["params.prompt"]}')

runs recorded: 2
  fail  faithfulness=0.0  prompt=Do you price match other shops?
  pass  faithfulness=1.0  prompt=How do I get a refund?

What the capstone put together

• Structured. The model name came from config and the prompt from the library, so the assistant's behaviour is in files you version, not scattered through the code.
• Tracked. Each answer is one MLflow run with its prompt, model, score and the answer text saved, so the record outlives the moment.
• Evaluated. A plain-Python faithfulness score and a pass-or-fail gate ran on every answer, and the failure mode shows the gate catching an ungrounded reply rather than letting it through.

Where this assembled shape fits

• A pre-ship gate, where every answer in an evaluation set is scored and logged, and a fail blocks the change.
• A regression check, where the tracked runs show whether a prompt or model change moved the scores.
• A handover, where the logged runs and their saved answers show a teammate exactly how the assistant did.

Related

• Previous: Observability in production
• Next: Maintenance and drift after you ship
• Reference: Logging a GenAI run to MLflow
• Reference: The app entry point and its API