Data validation

Data validation is the stage that checks an incoming file against a written schema, the columns it must carry and the dtype each one must have, and records the result before any training code touches the data.

Last updated: 19 Oct, 2026 · MLOps

The Data ingestion stage wrote three files and said nothing about what is in them. A column renamed upstream, a score column that arrives as text, a file that lost a column in an export: each of those reaches the training code as a confusing error deep inside a transformer, or worse, as a model that trains on the wrong thing. Validation turns that into one honest answer at the top of the run.

The end-to-end NLP project is where this stage is built on video. There the check is that the files the training step expects are present, and the result goes into a status file the pipeline reads. The video names the next step out loud, checking the columns and their types, and that is the check this stage makes.

Writing the schema as a YAML file

The schema is a file, not an assertion buried in code, so it is read by a person and edited without touching Python. Each key is a column the data must carry, and each value is the dtype pandas should report for it.

COLUMNS:
  gender: object
  race_ethnicity: object
  parental_level_of_education: object
  lunch: object
  test_preparation_course: object
  math_score: int64
  reading_score: int64
  writing_score: int64

A suite library is the other way to do this. great-expectations rewrote its API for version 1.0 and is on 1.24.0 today, so the 0.x pattern in older tutorials no longer exists. A typed check of your own is a dozen lines, reads as ordinary Python, and has no second API to keep up with.

The clip's check is that the expected files arrived, written out as a status line; this one checks the columns and dtypes of stud.csv against schema.yaml and writes a JSON report, and the stop-the-run-on-failure rule is the same.

Checking the columns against the schema

The component takes its config the same way the ingestion stage does. The first half of the check is set arithmetic: which expected columns are not here, and which columns are here that the schema never mentioned.

class DataValidation:
    def __init__(self, config: DataValidationConfig):
        self.config = config

    def validate(self, df: pd.DataFrame) -> dict:
        """Check the columns and dtypes the next stage assumes are there."""
        expected = self.config.expected_columns

        # set arithmetic both ways: what the schema wants, what the file brought
        missing = [c for c in expected if c not in df.columns]
        extra = [c for c in df.columns if c not in expected]

Checking the dtype of every column

The second half compares str(df[c].dtype) with the string in the schema, for the columns that are present. A score column arriving as float64 or object is caught here rather than by the model.

        # a score column arriving as text or as a decimal is caught here
        wrong_dtype = {
            c: str(df[c].dtype)
            for c, want in expected.items()
            if c in df.columns and str(df[c].dtype) != want
        }

        report = {
            "passed": not (missing or wrong_dtype),
            "missing_columns": missing,
            "extra_columns": extra,
            "wrong_dtype": wrong_dtype,
            "rows": int(len(df)),
        }

The report is saved as JSON and logged. A missing column or a wrong dtype fails the run; an unexpected extra column is recorded but allowed, because a source that adds a column has not broken anything the pipeline uses.

        save_json(self.config.status_file, report)
        logger.info("validation passed=%s", report["passed"])
        return report

The check as one function

The same check, written as a single function reading the schema from disk, so the two runs below each stand on their own.

import yaml


def validate(df, schema_path="schema.yaml"):
    """The component's check as one function: columns first, then dtypes."""
    # the schema is a file, so a column can be added without touching this code
    expected = yaml.safe_load(open(schema_path))["COLUMNS"]

    missing = [c for c in expected if c not in df.columns]
    extra = [c for c in df.columns if c not in expected]
    wrong_dtype = {c: str(df[c].dtype) for c, want in expected.items()
                   if c in df.columns and str(df[c].dtype) != want}

    return {"passed": not (missing or wrong_dtype), "missing_columns": missing,
            "extra_columns": extra, "wrong_dtype": wrong_dtype, "rows": int(len(df))}

Validating the ingested file

Point it at the copy the ingestion stage wrote. This is the answer a healthy run gives.

import json
import pandas as pd
from validate import validate

df = pd.read_csv("artifacts/data_ingestion/stud.csv")
print(json.dumps(validate(df), indent=4))

{
    "passed": true,
    "missing_columns": [],
    "extra_columns": [],
    "wrong_dtype": {},
    "rows": 1000
}

• passed is the field the pipeline branches on, and the three lists underneath it say why, so a failure does not need a debugger to explain.
• Writing this to artifacts/data_validation/status.json leaves a record of what the data looked like on the day the model was trained.
• rows is carried along because a file with the right columns and a tenth of the usual rows is also a problem worth seeing.

A file that fails the check

Break the frame in the two ways that happen in practice: a column dropped by an export, and a score column that arrives as a decimal. An extra column goes in as well, to show that one does not fail the run.

import json
import pandas as pd
from validate import validate

# an export that dropped a column, retyped another and added one of its own
broken = df.drop(columns=["lunch"])
broken["reading_score"] = broken["reading_score"].astype(float)
broken["extra_col"] = 1

print(json.dumps(validate(broken), indent=4))

{
    "passed": false,
    "missing_columns": [
        "lunch"
    ],
    "extra_columns": [
        "extra_col"
    ],
    "wrong_dtype": {
        "reading_score": "float64"
    },
    "rows": 1000
}

• missing_columns names lunch, so the failure points at one column instead of at the transformer that would have raised later.
• wrong_dtype reports float64 for reading_score, which the schema says should be int64. The model would have trained happily on it, which is what makes a silent type change worth catching.
• extra_col lands in extra_columns and passed stays false only because of the first two: the run is stopped by what is missing or mistyped, not by what was added.

A file check against a typed schema check

Where a validation stage earns its place

• A scheduled retrain on fresh data, where nobody is watching the run and the only thing standing between a changed export and a bad model is this check.
• A handover between teams, where schema.yaml is the contract the data producer can read without opening any Python.
• An incident after the fact, where status.json from each run says whether the data was already wrong on the day.

Related

• Previous: Data ingestion
• Next: Data transformation with ColumnTransformer
• Reference: pandas dtypes in the user guide