MLOpsThe MLOps lifecycle · MLflow 3.17 · DVC 3.67 · FastAPI · Docker · GitHub Actions · Evidently · Python 3.11+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
50 small wins to finish your pathNext lesson →

The signals to monitor

A monitoring signal is one number, read off the running service on a schedule, that tells you something about whether it is still working. A model service needs three kinds: what the data looks like, what the model is getting right, and whether the service is answering at all.

Last updated: 19 Oct, 2026 · MLOps

Data drift with Evidently gave one of those numbers. A drifted column on its own does not tell you to act, and a service can also fail in ways no drift report sees. This lesson is the whole board, and what each number is worth.

Grouping the signals by what they answer

The useful split is not data against model; it is how long you wait for the answer. Two of the three groups can be read the moment traffic arrives. The middle one waits for the truth, and the wait is why drift monitoring exists at all.

A board of nine monitoring signals in three groups. The first group, the data coming in, is answered immediately and needs no labels: data drift, each input column's distribution against the reference window; prediction drift, the distribution of what the model is predicting; and schema failures, requests rejected before the model is reached. The second group, the model's answers, needs labels and so arrives late: error rate as MAE or r2 once the true value is known; target drift, the distribution of the true values themselves; and concept drift, where the same inputs now map to different outcomes. The third group, the service around it, is answered immediately from the server: latency as p50 and p95 of a request rather than the average, failed requests meaning 5xx responses and unhandled errors, and queue depth, the work waiting, which is the first sign of saturation. A note says only the middle group tells you the model is wrong, while the other two tell you it is about to be, or that nobody is reaching it.
Model quality, error distribution, error normality and bias on a decayed week · from the Evidently AI Tutorial-Open Source ML Models Monitoring and Observability · 28:14 to 29:52

The clip reads these signals off Evidently 0.4's regression report, which 0.7 rebuilt and reaches differently; the signals it names, model quality, predicted against actual, the error distribution, error normality and bias, are the ones this lesson lists.

What each signal means

SignalWhat it isWhen you can read it
Data driftThe distribution of an input column against the reference windowImmediately, from the requests
Prediction driftThe distribution of the model's own output against the referenceImmediately, from the responses
Schema failuresRequests refused because a field is missing or out of rangeImmediately, from the API
Error rateMAE, RMSE or r2 of predictions against the true valuesOnce the true value is recorded
Target driftThe distribution of the true values themselvesOnce the true value is recorded
Concept driftThe same inputs now map to different outcomesOnly by comparing error over time
LatencyHow long a request takes, read as p50 and p95Immediately, from the server
Failed requests5xx responses and unhandled exceptionsImmediately, from the server
Queue depthHow much work is waiting to be servedImmediately, from the server

Prediction drift deserves its own line, because it is the cheapest early warning there is. You do not need a true value to notice that a model which used to predict a mean of 64 is now predicting a mean of 73. Nothing about the model changed, so the inputs did.

The schema the service checks every request against

A request that fails validation is a signal, not an error to hide. The same Pydantic model the API uses is reusable here: it says which fields must be there and that the two scores must sit between 0 and 100.

python
from pydantic import BaseModel, Field, ValidationError

class StudentRecord(BaseModel):
    gender: str
    race_ethnicity: str
    parental_level_of_education: str
    lunch: str
    test_preparation_course: str
    reading_score: int = Field(ge=0, le=100)   # a mark outside this range is not a mark
    writing_score: int = Field(ge=0, le=100)
python
def accepted(record):
    """True when the record passes the schema the API checks every request against."""
    try:
        StudentRecord(**record)
        return True
    except ValidationError:
        return False

The model and the traffic to watch

The model here is the one from Why a live model decays, fitted on the students who did no preparation course, and the traffic is the group that arrived after it. Three of those records carry a reading score of 150, which is the kind of thing a real feed sends on a bad day.

python
model = fit_forest(ref)

# the traffic to watch, with three records that will not pass the schema
live = cur.copy()
live.loc[live.index[:3], "reading_score"] = 150

Reading a day of traffic as five numbers

Everything above the last line can be read today. The last line is the one that waits for the marks to come back, and it is printed here only because this dataset happens to carry them.

Example
from sklearn.metrics import mean_absolute_error

# every record is checked before it is scored, the way the API checks it
features = live.drop(columns=["math_score"])
passed = features.apply(lambda row: accepted(row.to_dict()), axis=1)
predictions = model.predict(features[passed])

print("requests             ", len(features))
print("served               ", int(passed.sum()))
print("rejected             ", int((~passed).sum()))
print("mean prediction      ", round(float(predictions.mean()), 2))
print("reference mean       ", round(float(model.predict(ref.drop(columns=["math_score"])).mean()), 2))
print("MAE once marks arrive", round(mean_absolute_error(live.math_score[passed], predictions), 4))

What those five numbers say

  • Three rejected out of 358 is the schema doing its job. A rejection is cheap and visible; the same three records reaching the model would have produced three confident wrong answers instead.
  • The mean prediction moved by about nine marks, from the reference window to this one. No true value was needed to see it, which makes prediction drift the fastest signal on the board.
  • The error rate is the slow one. It is the number that proves the model is wrong rather than merely being asked something new, and it exists only because the marks happen to be in this dataset.
  • Counts matter as much as rates. Served and rejected together say how much traffic there was; a rate on its own hides a day when nobody called the service.

Reading latency as percentiles, not averages

  • An average hides the tail. If 99 requests take 20 ms and one takes 4 seconds, the average looks fine and one user waited 4 seconds. p95 and p99 are what a user feels.
  • Model latency is not request latency. Loading the frame, transforming it and running the forest is one part; the network, the gateway and the queue are the rest. Time them separately or the number tells you nothing about where to look.
  • Queue depth leads latency. Work piling up is visible before response times rise, which makes it the signal worth alerting on when traffic grows.
Watch out. Alerting on every signal is how a team learns to ignore alerts. Pick one page-worthy number per group: failed requests for the service, the drift share for the data, the error rate for the model. Everything else belongs on a dashboard somebody reads on a Monday.
Try it yourself
  • Corrupt ten records instead of three and watch the rejected count rise while the mean prediction barely moves.
  • Set one record's lunch to a value the training data never had, and check that it is served rather than rejected: the schema only knows types and ranges, not categories.
  • Print the predictions' standard deviation next to their mean, and decide whether a narrowing spread would worry you.
PreviousData drift with Evidently

Little by little, you're building something great.