Count and frequency encoding

Count encoding replaces each category with how many rows carry it, and frequency encoding replaces it with that count divided by the number of rows, so a column of any number of levels becomes one numeric column.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

One hot encoding with many categories cut 44 levels down to ten columns. This encoder handles the same column in one, and pays for it in a way worth understanding before using it.

The clip lists the advantages and then answers the question a viewer asks about two categories with the same count, which is the weakness of the technique.

The clip works on the country column of a census dataset with 42 levels. The page uses the Mercedes X2 column with 44, so the counts are the ones the previous lesson plotted.

Syntax: a value-count dictionary and a map.

counts = series.value_counts().to_dict()                 # label -> rows
shares = series.value_counts(normalize=True).to_dict()   # label -> share of rows
encoded = series.map(counts)

Counting the levels and mapping them back

The counts as a dictionary

value_counts returns a Series indexed by label. to_dict turns it into the lookup that map wants.

counts = df["X2"].value_counts().to_dict()   # {'as': 1659, 'ae': 496, ...}

Two new columns, in the frame

normalize=True gives the share instead of the count. The two columns hold the same information up to a constant divisor, so a model sees no difference between them, and the share is the one that survives a change in dataset size.

df["X2_count"] = df["X2"].map(counts)
df["X2_freq"] = df["X2"].map(shares)

44 levels in one column

import pandas as pd

URL = ("https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/"
       "master/mercedes.csv")
df = pd.read_csv(URL, usecols=["X1", "X2"])

counts = df["X2"].value_counts().to_dict()
shares = df["X2"].value_counts(normalize=True).to_dict()
df["X2_count"] = df["X2"].map(counts)
df["X2_freq"] = df["X2"].map(shares).round(6)

print(df.head(6).to_string(index=False))
print("columns now:", df.shape[1], " one hot would have added",
      df["X2"].nunique())

X1 X2  X2_count  X2_freq
 v at         6 0.001426
 t av         4 0.000950
 w  n       137 0.032549
 t  n       137 0.032549
 v  n       137 0.032549
 b  e        81 0.019244
columns now: 4  one hot would have added 44

What the encoded column holds

• Two source columns plus two encoded ones. Where one hot added 44 columns for X2, this adds one, whatever the cardinality is.
• Row 2 reads 137 and 0.032549. Level n appears in 137 of the 4209 rows, which is 3.25% of them. The two columns are the same fact in two units.
• Rows 0 and 1 read 6 and 4. Levels at and av, the rare ones the top-ten trick merged into an all-zero row. Here they keep separate values.
• No order was invented and no column was added. The number means something real about the level: how common it is.

Two levels with the same count

The encoding is a lookup from label to count, so two labels that appear the same number of times get the same number, and nothing downstream can tell them apart again. On a real column this is not a hypothetical.

import pandas as pd

URL = ("https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/"
       "master/mercedes.csv")
x2 = pd.read_csv(URL, usecols=["X2"])["X2"]
counts = x2.value_counts()

shared = counts.value_counts()
shared = shared[shared > 1].sort_index()
for rows, levels in shared.items():
    print(f"count {rows:4d}  shared by {levels} levels:",
          list(counts[counts == rows].index))
print("levels with a count nobody else has:",
      int((counts.value_counts() == 1).sum()), "of", len(counts))

count    1  shared by 8 levels: ['am', 'j', 'af', 'l', 'aa', 'c', 'o', 'ar']
count    4  shared by 3 levels: ['av', 'ah', 'p']
count    5  shared by 3 levels: ['al', 'an', 'q']
count    6  shared by 2 levels: ['at', 'h']
count   11  shared by 2 levels: ['ap', 'y']
count   19  shared by 2 levels: ['ag', 'z']
count   25  shared by 2 levels: ['k', 'i']
levels with a count nobody else has: 22 of 44

What the collisions cost

• Only 22 of the 44 levels keep a count nobody else has. The other 22 share theirs with at least one other level, so half the column loses its identity to a collision.
• k and i both appear 25 times. After count encoding both are the number 25, and the model cannot separate two levels that may behave completely differently.
• at and h both appear 6 times. Same collision, further down the tail.
• Eight levels appear once each. All eight collapse to the single value 1, which merges the eight rarest levels of the column into one.
• The collisions cluster in the tail. The frequent levels have counts of their own, so the encoding keeps the head of the distribution and blurs the tail, which is the same trade the top-ten trick makes by a different route.

Two ways to break this encoding

Both failures are silent, which is why they are worth causing deliberately once. Mapping a column that is already numeric finds no keys, and mapping a level the counts were never built from finds no key either.

import pandas as pd

train = pd.Series(["as", "ae", "as", "n"])
counts = train.value_counts().to_dict()
encoded = train.map(counts)
print("encoded once :", encoded.tolist())
print("encoded twice:", encoded.map(counts).tolist())

test = pd.Series(["as", "zz", "n"])        # zz was never in the training rows
print("test encoded :", test.map(counts).tolist())
print("filled with 0:", test.map(counts).fillna(0).tolist())

encoded once : [2, 1, 2, 1]
encoded twice: [nan, nan, nan, nan]
test encoded : [2.0, nan, 1.0]
filled with 0: [2.0, 0.0, 1.0]

Why both results are NaN

• Encoding twice gives all NaN. The second map looks up the numbers 2, 1, 2 and 1 in a dictionary whose keys are the strings as, ae and n. Nothing matches, so every row becomes NaN, and running the cell a second time in a notebook is enough to cause it.
• The unseen level zz gives NaN. The counts were built from the training rows and zz was not among them, so the lookup fails for that row only.
• A NaN here is correct and still has to be decided. fillna(0) says an unseen level appeared zero times in training, which is true and is usually the right filling.
• Neither case raised. A model fitted on a column of NaN fails much later, somewhere that gives no hint about which map call caused it.

Count encoding vs one hot

Where the count is a real feature

• Popularity as a signal. How often a product, a city or a browser appears is often predictive on its own, and this encoding hands the model that fact directly.
• A column with hundreds of levels feeding a tree. Gradient boosting splits on the number, so a single count column lets it separate frequent levels from rare ones in one split.
• Alongside another encoding. The count column and the top-ten indicators answer different questions and can both sit in the frame.

Related

• Previous: Label encoding
• Next: Target guided ordinal encoding
• See also: One hot encoding with many categories
• Reference: pandas.Series.value_counts