EDA and Feature Engineeringpandas 3.0 · scikit-learn 1.9 · seaborn 0.13 · Python 3.12+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
72 small wins to finish your pathNext lesson →

One hot encoding with many categories

One hot encoding a high cardinality column adds one column per level, so a column with dozens of levels turns a narrow frame into a wide and mostly empty one.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

One hot encoding cost one extra column for Sex and two for Embarked. This lesson runs the same call on a column with 44 levels and on the answer competition winners found for it.

Measuring the cost on a 44 level column

Two columns of a car dataset

The Mercedes-Benz manufacturing dataset has 378 columns; usecols loads two of them. X1 and X2 are anonymised production settings, which makes them the purest example of a nominal column with no order available anywhere.

python
df = pd.read_csv(URL, usecols=["X1", "X2"])   # 2 of the 378 columns

Counting the levels before encoding

nunique on each column is the one call worth making before any one hot encoding at all. It is the number of columns the encoding is about to add.

python
for column in df.columns:
    print(column, df[column].nunique(), "levels")

71 columns out of 2

ExampleMercedes X1 and X2, one hot encoded in full
import pandas as pd

URL = ("https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/"
       "master/mercedes.csv")
df = pd.read_csv(URL, usecols=["X1", "X2"])   # 2 of the 378 columns

print("shape:", df.shape)
for column in df.columns:
    print(column, df[column].nunique(), "levels")

print("get_dummies            :", pd.get_dummies(df, dtype=int).shape)
print("get_dummies drop_first :",
      pd.get_dummies(df, drop_first=True, dtype=int).shape)

What 71 columns cost

  • Two columns became 71. 27 levels in X1 and 44 in X2. With drop_first it is 69, which is a saving of 2 and not a solution.
  • Each new column is almost all zeros. A level that appears in 1 row of 4209 gives a column with one 1 in it, which no model can learn anything from.
  • The frame is 4209 rows wide by 71 columns. Real catalogues have thousands of levels, and the same call on a postcode column produces a frame with more columns than rows.
  • A tree cannot use them either. Splitting on a column that is 1 in four rows out of 4209 separates four rows, which is the shape of overfitting rather than of a useful rule.

How the levels are distributed

The counts are what makes the trick work. Plotting all 44 levels of X2 in order of frequency shows where the information in the column lives.

ExampleAll 44 levels of X2 by how often they appear
import matplotlib
matplotlib.use("svg")
import matplotlib.pyplot as plt
import pandas as pd

URL = ("https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/"
       "master/mercedes.csv")
df = pd.read_csv(URL, usecols=["X1", "X2"])
counts = df["X2"].value_counts()

print("top 5 :", counts.head(5).to_dict())
print("last 5:", counts.tail(5).to_dict())
print("levels appearing once:", int((counts == 1).sum()))
print("share of rows in the top 10:",
      round(counts.head(10).sum() / len(df) * 100, 1), "%")

fig, ax = plt.subplots(figsize=(7.5, 3))
colours = ["#9370DB"] * 10 + ["#c9c9d6"] * (len(counts) - 10)
ax.bar(range(len(counts)), counts.values, color=colours)
ax.set_xlabel("the 44 levels of X2, most frequent first")
ax.set_ylabel("rows")
fig.savefig("chart.svg")
A bar chart of the 44 levels of the X2 column, ordered from most to least frequent. The first bar reaches 1659 rows, the second 496, and the bars fall away quickly; the first ten bars are purple and the remaining 34 are grey and barely visible above the axis.

Reading the long tail

  • One level covers 1659 of 4209 rows. The level named as covers 39% of the dataset on its own.
  • The top ten levels cover 89.2% of the rows. The other 34 levels share the remaining 11%.
  • Eight levels appear exactly once. Eight of the 44 columns one hot encoding would add hold a single 1 each.
  • The shape is the usual one. Most real categorical columns look like this, which is why a rule that keeps the head and groups the tail works so widely.

Encoding only the ten most frequent levels

The trick is to make an indicator column for each of the ten most frequent levels and for no other level. A row whose level is outside the ten comes out as zeros across all ten, which is the same pattern a dropped level produces. It comes from the write-up of the KDD Cup 2009 Orange Challenge winners, and it costs ten columns per feature instead of 44.

ExampleTen indicator columns for the ten most frequent levels of X2
import numpy as np
import pandas as pd

URL = ("https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/"
       "master/mercedes.csv")
df = pd.read_csv(URL, usecols=["X1", "X2"])

top_10 = list(df["X2"].value_counts().head(10).index)
print("top 10:", top_10)

for label in top_10:
    df["X2_" + label] = np.where(df["X2"] == label, 1, 0)

shown = ["X2", "X2_as", "X2_ae", "X2_n", "X2_e"]   # 4 of the 10 new columns
print(df[shown].head(6).to_string(index=False))
print("frame shape now:", df.shape)
covered = df["X2"].isin(top_10)
print("rows with a 1 somewhere:", int(covered.sum()),
      " rows that are all zero:", int((~covered).sum()))

What the ten columns bought

  • Ten columns instead of 44. The frame went from 2 columns to 12, and 34 near-empty columns were never created.
  • 3754 rows carry a 1 and 455 do not. Those 455 rows hold one of the 34 rare levels, and all ten indicators are 0 for them.
  • Rows 0 and 1 are both all zero. Their levels are at and av, which appear 6 and 4 times in 4209 rows. Row 2 is level n, which is in the top ten, so its X2_n column is 1.
  • Every rare level now looks the same. at and av are indistinguishable after this encoding. That is the price, and on a level that appears 4 times it is a price worth paying.

A level that arrives for the first time after the model ships gets the same all-zero pattern, with no special handling, which is the second reason this trick is used. The scikit-learn version of the same idea is OneHotEncoder(max_categories=10) or min_frequency=, and it keeps the chosen levels inside the fitted object.

Full one hot vs the top ten

Every levelTop ten only
Columns added for X24410
Rare levelsone column each, nearly emptymerged into the all-zero pattern
Rows that are all zero0455 of 4209
A level seen for the first time laterno column exists for itall ten are 0, which is a pattern the model has seen
Information keptall of itwhich of the common levels, and nothing finer

Where a wide column turns up

  • Postcode, city and country. Hundreds to thousands of levels, with most of the rows in a handful of them.
  • Product, SKU and brand. A catalogue grows, so the level set is not even fixed.
  • Free-text categories. A column people typed into carries misspellings and case variants as separate levels, which inflates the count before any real diversity does.
Watch out. Pick the ten levels from the training rows only. Counting on the whole dataset lets the test rows decide which levels get a column, and the score that follows is better than the one the same code produces in production.
Try it yourself
  • Change head(10) to head(3) and count how many rows come out all zero.
  • Run the same encoding on X1, which has 27 levels, and compare the share of rows the top ten cover.
  • Replace the loop with OneHotEncoder(max_categories=10, handle_unknown="infrequent_if_exist") and print get_feature_names_out().
PreviousOne hot encoding

Slow is fine. Stopping is the only problem.