Experiment tracking

Experiment tracking is a record of every training run with the settings it used and the scores it earned, so a question like which configuration was best is answered from the record rather than from memory.

Last updated: 19 Oct, 2026 · MLOps

The prediction pipeline turned one trained model into predictions. Training it once is never what happens: you change the depth of the forest, the number of trees, the columns you keep, and each change produces a slightly different model. The video calls this experiment tracking and model registration, and names MLflow as the tool for both. The video logs its runs to a hosted server from the first line; these pages log to a folder on your own machine, and the hosted server is the subject of A remote tracking server at the end of this part.

The MLflow UI was redesigned for 3.x, so the experiment list on screen is laid out differently from the one you will open; the loop it shows, change a setting and run again and a new run appears, is unchanged.

What one run holds

MLflow stores one run per execution. A run holds the parameters you chose, the metrics you measured, the tags you labelled it with, and the files it produced, including the model. Runs are grouped under a named experiment, which is how the runs of one project stay together.

Installing MLflow

MLflow is the one new library this part needs. Pin the version, because the file-store behaviour below is the behaviour of this release.

pip install "mlflow==3.17.1"

Pointing MLflow at a local store

Syntax: A tracking URI tells MLflow where to write. A local file store lives in an mlruns folder beside your code, so no account and no network are needed. One line before it is required rather than decoration: MLflow 3.17 refuses a file store unless MLFLOW_ALLOW_FILE_STORE is set.

import os, shutil

os.environ["MLFLOW_ALLOW_FILE_STORE"] = "true"   # 3.17 refuses a local file store without this
shutil.rmtree("mlruns", ignore_errors=True)      # each lesson starts from an empty store
os.makedirs("mlruns/.trash", exist_ok=True)      # MLflow keeps deleted runs in .trash

import mlflow
mlflow.set_tracking_uri("file:./mlruns")
mlflow.set_experiment("student-performance")

Loading the student data and splitting it

The run needs something to track. This part fits the student-performance forest in one file, with the same split and the same seed the trainer stage used, so a score here is comparable with a score there.

import pandas as pd
from sklearn.model_selection import train_test_split

URL = "https://raw.githubusercontent.com/krishnaik06/mlproject/main/notebook/data/stud.csv"

# the same 80/20 split and seed as the model trainer, so the scores match that stage
train, test = train_test_split(pd.read_csv(URL), test_size=0.2, random_state=42)
Xtr, ytr = train.drop(columns=["math_score"]), train["math_score"]
Xte, yte = test.drop(columns=["math_score"]), test["math_score"]

Building the preprocessor

The five text columns are one-hot encoded and the two score columns are scaled, which turns seven feature columns into nineteen. Fitting happens on the training rows only.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# five text columns are one-hot encoded, the two score columns are scaled
cats = Xtr.select_dtypes(include="object").columns.tolist()
nums = Xtr.select_dtypes(exclude="object").columns.tolist()

pre = ColumnTransformer([
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False))]), cats),
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), nums)])

Fitting one configuration

fit takes the one setting this part varies, the depth of the trees, and hands back the fitted forest with its r2, MAE and RMSE. Everything tracked below comes out of this function.

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

A, B = pre.fit_transform(Xtr), pre.transform(Xte)   # fit on train, transform the test set

def fit(max_depth):
    """Train one configuration and hand back the model with its three scores."""
    model = RandomForestRegressor(n_estimators=100, max_depth=max_depth, random_state=42)
    p = model.fit(A, ytr).predict(B)
    return model, {"r2": round(r2_score(yte, p), 4),
                   "mae": round(mean_absolute_error(yte, p), 4),
                   "rmse": round(mean_squared_error(yte, p) ** 0.5, 4)}

Recording the first run

mlflow.start_run opens a run, and the with block closes and writes it. log_metric records one measured number. search_runs reads the experiment back as a DataFrame, which is the proof that the run survived the end of the process.

# one tracked run: the model is fitted inside it and the score it earned is recorded
model, scores = fit(max_depth=12)

with mlflow.start_run(run_name="depth-12"):
    mlflow.log_metric("r2", scores["r2"])

runs = mlflow.search_runs(experiment_names=["student-performance"])
print("runs in the experiment:", len(runs))
print("run name              :", runs.loc[0, "tags.mlflow.runName"])
print("r2 recorded           :", runs.loc[0, "metrics.r2"])

runs in the experiment: 1
run name              : depth-12
r2 recorded           : 0.8485

Reading what the run kept

• The experiment holds exactly one run, created by the with block, and it is on disk in mlruns rather than in the memory of the process that made it.
• The run name is a label you chose, depth-12, which is how you recognise a run later without reading its id.
• The r2 came out of the fit, not out of a note in a notebook cell, so the number in the store is the number the model earned.
• Nothing needed an account. The store is a folder, which is why this works offline and why you can delete it and start again.

A notebook record against a tracked record

Where you start tracking

• Any sweep over a setting, where three forests with different depths are three runs you compare instead of three cells you scroll back through.
• A shared project, where a teammate needs to see what was tried, with the numbers attached.
• A retrain, where next month's model has to be compared against the one in production.

Related

• Previous: The prediction pipeline
• Next: Logging params and metrics
• Reference: MLflow Tracking