EDA and Feature Engineeringpandas 3.0 · scikit-learn 1.9 · seaborn 0.13 · Python 3.12+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
72 small wins to finish your pathNext lesson →

Project: building the features

The feature pass turns 79 raw columns into 82 numbers between 0 and 1, with every value it fills, groups or codes taken from the 1,314 training houses.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

Project: the EDA pass ended with a table of decisions. This lesson carries them out, in an order that matters: a flag has to be written before the gap is filled, and a gap has to be filled before a year becomes an age.

How 79 raw columns become 82 numbers. The 36 number columns gain three flags recording whether LotFrontage, MasVnrArea or GarageYrBlt was missing, have their gaps filled with the training median, turn three year columns into ages by subtracting from YrSold, and take a log of four skewed columns, which gives 39 number columns. The 43 text columns turn a gap into the label Missing, group labels under one percent of the training rows into one Rare_var label, give each label a code ordered by the mean training price of its rows, and code an unseen label as minus one. Both blocks go into a MinMaxScaler fitted on train, giving 82 columns between 0 and 1.

Writing a flag before filling the gap

Three numeric columns have gaps. Filling them hides which rows were empty, and whether a house had its frontage recorded may itself carry information, so a 0 or 1 column is written first. The function reads one row at a time and learns nothing, which is why it can run on both halves.

python
NANCOLS = ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']
YEARS = ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']
LOGS = ['LotFrontage', 'LotArea', '1stFlrSF', 'GrLivArea']
python
def flags(frame):
    frame = frame.copy()
    for c in NANCOLS:
        frame[c + 'nan'] = frame[c].isnull().astype(int)
    return frame

Filling with the training median

train[numbers].median() is one number per column, learned once. Both halves are filled with it. Writing a whole block of columns at once, train[numbers] = train[numbers].fillna(medians), assigns back into the frame; the in-place form through a single column does nothing under pandas 3, which Copy-on-write and chained assignment shows.

python
medians = train[numbers].median()
train[numbers] = train[numbers].fillna(medians)
test[numbers] = test[numbers].fillna(medians)

Turning years into ages and taking logs

A year of construction is a label on a timeline; the age of the house at the sale is a quantity. Subtracting from YrSold turns one into the other. The four skewed measurements take a log at the same time. Both are fixed formulas, so they live in one function with no fitting in it.

python
def ages_and_logs(frame):
    frame = frame.copy()
    for c in YEARS:
        frame[c] = frame['YrSold'] - frame[c]
    for c in LOGS:
        frame[c] = np.log(frame[c])
    return frame

Giving the gaps in text columns their own label

No pool, no alley and no fence are facts about the house, so the gap becomes the label Missing rather than the most common grade. Missing as its own category covers when that is the right call.

python
train, test = ages_and_logs(train), ages_and_logs(test)
train[words] = train[words].fillna('Missing')
test[words] = test[words].fillna('Missing')

Grouping rare labels and coding by the training price

Any label held by fewer than one percent of the training houses joins one Rare_var group, which stops a label with four houses getting a code of its own. Then each surviving label gets an integer, ordered by the mean log price of the training houses that carry it, so the codes run from cheapest to dearest. A label that only the held back houses carry maps to nothing, and is coded -1.

python
codes = {}
for c in words:
    share = train[c].value_counts(normalize=True)
    keep = share[share > 0.01].index
    train[c] = np.where(train[c].isin(keep), train[c], 'Rare_var')
    test[c] = np.where(test[c].isin(keep), test[c], 'Rare_var')
    order = y_train.groupby(train[c]).mean().sort_values().index
    codes[c] = {label: i for i, label in enumerate(order)}
    train[c] = train[c].map(codes[c]).astype(float)
    test[c] = pd.Series(test[c], index=test.index).map(codes[c]).fillna(-1).astype(float)

Scaling every column into the same range

MinMaxScaler is fitted on the training block and applied to both. set_output(transform='pandas') keeps the column names, so the result is still a frame you can read.

python
features = numbers + words
scaler = MinMaxScaler().set_output(transform='pandas').fit(train[features])
train_ready = scaler.transform(train[features])
test_ready = scaler.transform(test[features])

Running the whole pass

Example79 columns in, 82 scaled columns out
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

AMES = ("https://raw.githubusercontent.com/krishnaik06/"
        "Advanced-House-Price-Prediction-/master/train.csv")
houses = pd.read_csv(AMES)
y = np.log(houses['SalePrice'])
X = houses.drop(columns=['Id', 'SalePrice'])
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.1, random_state=0)

NANCOLS = ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']
YEARS = ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']
LOGS = ['LotFrontage', 'LotArea', '1stFlrSF', 'GrLivArea']

def flags(frame):
    frame = frame.copy()
    for c in NANCOLS:
        frame[c + 'nan'] = frame[c].isnull().astype(int)
    return frame

def ages_and_logs(frame):
    frame = frame.copy()
    for c in YEARS:
        frame[c] = frame['YrSold'] - frame[c]
    for c in LOGS:
        frame[c] = np.log(frame[c])
    return frame

train, test = flags(X_train), flags(X_test)
numbers = [c for c in train.columns if pd.api.types.is_numeric_dtype(train[c])]
words = [c for c in train.columns if pd.api.types.is_string_dtype(train[c])]
print("numeric", len(numbers), "text", len(words),
      "total", len(numbers) + len(words))

medians = train[numbers].median()
print("medians learned on train:", medians[NANCOLS].to_dict())
train[numbers] = train[numbers].fillna(medians)
test[numbers] = test[numbers].fillna(medians)

train, test = ages_and_logs(train), ages_and_logs(test)
train[words] = train[words].fillna('Missing')
test[words] = test[words].fillna('Missing')

codes = {}
for c in words:
    share = train[c].value_counts(normalize=True)
    keep = share[share > 0.01].index
    train[c] = np.where(train[c].isin(keep), train[c], 'Rare_var')
    test[c] = np.where(test[c].isin(keep), test[c], 'Rare_var')
    order = y_train.groupby(train[c]).mean().sort_values().index
    codes[c] = {label: i for i, label in enumerate(order)}
    train[c] = train[c].map(codes[c]).astype(float)
    test[c] = pd.Series(test[c], index=test.index).map(codes[c]).fillna(-1).astype(float)

features = numbers + words
scaler = MinMaxScaler().set_output(transform='pandas').fit(train[features])
train_ready = scaler.transform(train[features])
test_ready = scaler.transform(test[features])
print("MSZoning code:", codes['MSZoning'])
print("train", train_ready.shape, "test", test_ready.shape)
print("nulls left:", int(train_ready.isnull().sum().sum()),
      int(test_ready.isnull().sum().sum()))
print(train_ready[['LotFrontage', 'YearBuilt', 'MSZoning',
                   'Neighborhood', 'LotFrontagenan']].head(3).round(4).to_string())
  • 39 numbers and 43 coded text columns. The three flags are what take the numeric side from 36 to 39, and nothing is dropped, so the total is 82.
  • The medians are the training ones. 69.0 feet of frontage, 0.0 square feet of masonry veneer and the year 1980 for a garage. Those three numbers now belong to the model and have to travel with it.
  • MSZoning codes run 0 to 4 with Rare_var at 0. The rare group turned out to be the cheapest, and RL, the common residential zone, sits at 3.
  • No nulls on either side. The test half is clean too, filled and coded with the training numbers, never with its own.
  • The first three rows are all between 0 and 1. LotFrontagenan is 0 for all three, so none of those houses was missing its frontage.

What each step learned, and from where

StepLearned from the training rowsApplied to the held back rows
flagsnothing, it reads one row at a timethe same rule
median fillone median per numeric columnthe same medians
ages and logsnothing, both are formulasthe same formulas
Missing labelnothing, the label is a constantthe same label
rare groupingwhich labels clear one percentthe same list, everything else grouped
target guided codesone integer per label, ordered by the training pricethe same codes, an unseen label becomes -1
MinMaxScalerthe minimum and maximum of all 82 columnsthe same formula, so a value may land outside 0 to 1

Where this pass is reused

  • Any wide table with mixed types. The shape of this pass, flags then fills then formulas then codes then scale, transfers to almost any tabular problem.
  • Any table where a gap means something. The flag plus the fill keeps both the value and the fact that it was missing, which one or the other alone throws away.
  • Any model that needs comparable columns. Lasso, the model this project ends with, penalises coefficients, so a column measured in thousands would be punished for its units.
Watch out. The order inside the loop matters. Grouping rare labels after computing the codes would give Rare_var no code at all, and taking the codes from the whole table instead of y_train is the second door in Data leakage, on a column the selection is about to rank highly.
Try it yourself
  • Change the rare threshold from 0.01 to 0.05 and print codes['Neighborhood']. Far more neighbourhoods collapse into one group.
  • Drop the three flag columns by emptying NANCOLS and print the shape. 79 columns instead of 82, and the model loses the fact that a value was missing.
  • Print test_ready.max() and count the columns above 1. Those are held back houses that went past the training maximum, which is what a fitted scaler is supposed to do.
PreviousProject: the EDA pass

This is what real progress feels like.