Why a live model decays

Model decay is the loss of accuracy a deployed model suffers when the records it scores stop looking like the records it was fitted on. The file on disk never changes. The world it answers questions about does, and the score falls without a single line of code being touched.

Last updated: 19 Oct, 2026 · MLOps

Secrets and what never goes in git finished the automation: a push builds the image and ships it. From here the service is running, and the work changes shape. Nobody is editing code any more; the job is watching a model that was right in July and is quietly less right in November.

Why a frozen model goes wrong

• The inputs move. A new intake of students, a new region, a new price band: the columns are the same, the distribution behind them is not. This is data drift, and the drift lesson measures it.
• The relationship moves. The same inputs start mapping to different outcomes, because a policy changed or a competitor appeared. This is concept drift, and no amount of input checking spots it without labels.
• A value shows up that was never in training. A category the encoder never saw becomes a row of zeros, and the model falls back on whatever else it has.
• An upstream feed changes quietly. A column arrives in a different unit or starts coming through empty, and the pipeline keeps running because nothing raised an error.

The student-performance data gives a clean version of the third and the first together. Splitting the dataset by test_preparation_course gives two populations: 642 students who did no preparation course, and 358 who completed one. Fit on the first group, serve the second, and nothing in the service is wrong; the records are.

Splitting the students into a before and an after

The dataset loads straight from the repo, so there is nothing to download by hand. The two lists name the text columns and the score columns, which is all the preprocessor needs to know.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# the course's dataset, read over the network so no file has to be fetched first
URL = "https://raw.githubusercontent.com/krishnaik06/mlproject/main/notebook/data/stud.csv"
df = pd.read_csv(URL)

# the five text columns and the two score columns, named once and reused
CAT = ["gender", "race_ethnicity", "parental_level_of_education", "lunch", "test_preparation_course"]
NUM = ["reading_score", "writing_score"]

Fitting the forest on one group only

fit_forest wraps the one-hot encoder, the scaler and the forest in a single sklearn Pipeline, with the settings the training pipeline uses: 100 trees, a depth of 12, seed 42. One object holds the preprocessing and the model, so there is no chance of scoring a frame the encoder never saw.

def fit_forest(frame):
    """Fit the course's preprocessor and forest on one slice of the students."""
    pre = ColumnTransformer([
        ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), CAT),
        ("num", StandardScaler(), NUM),
    ])
    forest = RandomForestRegressor(n_estimators=100, max_depth=12, random_state=42)
    return Pipeline([("pre", pre), ("rf", forest)]).fit(
        frame.drop(columns=["math_score"]), frame.math_score)

# the group the model learns from, and the group that turns up later
ref = df[df.test_preparation_course == "none"]
cur = df[df.test_preparation_course == "completed"]

Scoring the same model on both groups

The model is fitted once, on 80% of the reference group. Then the same fitted object scores two things: the fifth of the reference group it was held back from, and the whole of the new group. One of those is the number a notebook reports. The other is the number the service lives with.

from sklearn.metrics import mean_absolute_error, r2_score

# fitted once, on four fifths of the reference students, and never refitted after
train, held_out = train_test_split(ref, test_size=0.2, random_state=42)
model = fit_forest(train)

for name, group in [("held-out reference students", held_out), ("the new group", cur)]:
    truth = group.math_score
    pred = model.predict(group.drop(columns=["math_score"]))
    print(f"{name:28} r2 {r2_score(truth, pred):.4f}  MAE {mean_absolute_error(truth, pred):.4f}")

held-out reference students  r2 0.7817  MAE 5.1428
the new group                r2 0.7477  MAE 5.7959

Reading the two scores

• The held-out reference students are the notebook number. They were never trained on, but they come from the same population, and that is the number that gets written in the model card.
• The new group is the production number. Same file, same code, a different intake of students, and the error is most of a mark worse per prediction.
• The drop is small and that is the problem. Nothing crashed, no alert fired, and the service carried on answering. Decay rarely arrives as a failure; it arrives as a number nobody is looking at.
• One cause is visible in the data. Every record in the new group carries test_preparation_course set to completed, a value the encoder never saw during training, so it is encoded as all zeros.

A notebook model against a served model

Catching decay before the labels arrive

• True values are late, and sometimes never come. A predicted exam mark is checkable in weeks; a predicted churn is checkable in months. Waiting for labels to notice decay means months of wrong answers.
• The inputs are there immediately. Every request carries its own features, so the shift in the inputs can be measured the same day, long before any mark is recorded.
• That is why drift detection exists. It is the early signal: not proof the model is wrong, but evidence that it is being asked something new.

Related

• Previous: Secrets and what never goes in git
• Next: Data drift with Evidently
• See also: Retraining strategies