Data transformation with ColumnTransformer

ColumnTransformer is the scikit-learn object that applies a different pipeline to each group of columns and lays the results side by side in one matrix, so text columns and number columns are prepared together in a single fitted object.

Last updated: 19 Oct, 2026 · MLOps

Data validation confirmed the file carries the columns the schema asks for, with the types it asks for. Five of those columns are text and two are numbers, and a model takes neither as it stands. This stage turns seven feature columns into a numeric matrix, and saves the object that did it.

The video builds this in two halves: a pipeline for numeric columns and a pipeline for categorical columns, each a list of named steps, then one ColumnTransformer that names which columns each pipeline owns. Two details differ here. OneHotEncoder took an argument called sparse at the time; in scikit-learn 1.9.1 it is sparse_output. And the categorical imputer fills with the most frequent category rather than a constant, so a gap does not invent a category of its own.

Grouping the columns by kind

The component asks the frame which columns are text and which are not, instead of typing two lists. A column added upstream then lands in the right branch without an edit here.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

    def build_preprocessor(self, X: pd.DataFrame) -> ColumnTransformer:
        """One branch for text columns, one for numbers, fitted together."""
        cat = X.select_dtypes(include="object").columns.tolist()
        num = X.select_dtypes(exclude="object").columns.tolist()

The clip builds the two pipelines in a notebook; this lesson builds the same pair inside the transformation component, with the column lists read off the frame instead of typed out.

A pipeline for the text columns

Two named steps in order. The imputer fills a gap with the category seen most often, and the encoder turns each column into one new column per category. handle_unknown="ignore" means a category that was never in the training data comes through as all zeros rather than as an error at serving time, and sparse_output=False asks for a plain array.

        cat_steps = Pipeline([
            ("impute", SimpleImputer(strategy="most_frequent")),
            ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
        ])

A pipeline for the score columns

The numeric branch fills gaps with the median, which a few extreme scores cannot drag around, and then scales each column to a mean of zero and a standard deviation of one. Both are learned from the training rows.

        num_steps = Pipeline([
            ("impute", SimpleImputer(strategy="median")),
            ("scale", StandardScaler()),
        ])

        return ColumnTransformer([("cat", cat_steps, cat), ("num", num_steps, num)])

Fitting on the training rows and transforming the test rows

The run method is where the leakage rule is enforced. The preprocessor is fitted on the training features only, and the test features are passed through transform, which uses what the fit already learned and learns nothing new.

    def run(self, train: pd.DataFrame, test: pd.DataFrame):
        target = self.config.target_column
        Xtr, ytr = train.drop(columns=[target]), train[target]
        Xte, yte = test.drop(columns=[target]), test[target]

        pre = self.build_preprocessor(Xtr)
        Xtr_t = pre.fit_transform(Xtr)      # fit on train only
        Xte_t = pre.transform(Xte)          # the test set is only transformed

        save_object(self.config.preprocessor_file, pre)
        return Xtr_t, ytr, Xte_t, yte

Building the preprocessor and reading the shapes

The same build, flat, with the target column named. The printed shapes are the whole point of the stage: seven columns in, nineteen out, and the test set the same width as the training set.

target = "math_score"
Xtr, ytr = train.drop(columns=[target]), train[target]
Xte, yte = test.drop(columns=[target]), test[target]

cat = Xtr.select_dtypes(include="object").columns.tolist()
num = Xtr.select_dtypes(exclude="object").columns.tolist()

cat_steps = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False))])
num_steps = Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())])
pre = ColumnTransformer([("cat", cat_steps, cat), ("num", num_steps, num)])

Xtr_t = pre.fit_transform(Xtr)   # learns the categories, medians and scales from train
Xte_t = pre.transform(Xte)       # the test rows are only transformed
print(len(cat), "text and", len(num), "numeric ->", Xtr_t.shape, Xte_t.shape)

5 text and 2 numeric -> (800, 19) (200, 19)

• Both matrices are 19 columns wide, and they have to be: a model fitted on 19 inputs cannot score a matrix of a different width.
• The 800 and the 200 are the files ingestion wrote, carried through unchanged, so the test rows are still rows the fit never saw.
• fit_transform ran on the training features and transform on the test features. Calling fit_transform on the test set as well would recompute the medians, the scales and the category lists from the test rows, which is the leak this stage exists to prevent.

Where the nineteen columns come from

The width is not a setting. It is the number of categories the encoder found in each text column, plus the two scaled score columns. Saving the fitted object and loading it back shows that those counts travel with it.

import os
import joblib

os.makedirs("artifacts/data_transformation", exist_ok=True)
joblib.dump(pre, "artifacts/data_transformation/preprocessor.joblib")

# a fitted preprocessor carries the categories it saw, so it reloads ready to transform
reloaded = joblib.load("artifacts/data_transformation/preprocessor.joblib")
print({c: int(Xtr[c].nunique()) for c in cat})
print(reloaded.transform(Xte.head(2)).shape)

{'gender': 2, 'race_ethnicity': 5, 'parental_level_of_education': 6, 'lunch': 2, 'test_preparation_course': 2}
(2, 19)

• The five text columns carry 2, 5, 6, 2 and 2 categories, which is 17 one-hot columns, and the two score columns stay as two: 19 in total.
• The reloaded object transformed two fresh rows into the same 19 columns, because the fit stored the category lists, the medians and the scales inside it.
• That file is the one the serving path loads. A model shipped without the preprocessor that was fitted beside it has no way to turn a request into the matrix it was trained on.

fit_transform against transform

Where one fitted preprocessor matters

• Mixed data, where text and numbers need different treatment but one object has to describe the whole frame.
• Serving, where a single record has to be prepared exactly as the training rows were, with the same category order.
• A retrain, where the new preprocessor and the new model are saved as a pair and replace the old pair together.

Related

• Previous: Data validation
• Next: The model trainer
• Reference: ColumnTransformer in the scikit-learn documentation