EDA and Feature Engineeringpandas 3.0 · scikit-learn 1.9 · seaborn 0.13 · Python 3.12+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
72 small wins to finish your pathNext lesson →

Chi square for categorical features

The chi square test scores a feature by how far the counts it produces per class sit from the counts expected if the feature and the target were unrelated.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

Mutual information ranked columns by dependence of any kind. Chi square answers a narrower question with a statistical test behind it, and it is the standard filter for counts and categories.

The idea is a table. For each value of the feature and each class of the target, count the rows. Then work out what each count would be if the two were independent, which is the row total times the column total divided by the grand total. The statistic adds up (observed - expected)^2 / expected over every cell. A large value means the counts could not plausibly have come from an unrelated pair.

Syntax: chi2 returns two arrays, the statistic per column and the p value per column.

python
from sklearn.feature_selection import chi2, SelectKBest

statistic, p_value = chi2(X, y)                  # X must be non-negative
SelectKBest(score_func=chi2, k=10).fit(X, y)     # the same score, as a transformer

Scoring the mobile price columns

The mobile price dataset has 2000 phones, 20 columns describing each and a target with four price bands of 500 phones each. Every column is a non-negative count or measurement, which is what chi square needs.

python
import pandas as pd

MOBILE = ("https://raw.githubusercontent.com/krishnaik06/"
          "Feature-Engineering-Live-sessions/master/mobile_dataset.csv")
df = pd.read_csv(MOBILE)

X = df.drop(columns="price_range")
y = df["price_range"]
ExampleTwenty columns scored against four price bands
import pandas as pd

MOBILE = ("https://raw.githubusercontent.com/krishnaik06/"
          "Feature-Engineering-Live-sessions/master/mobile_dataset.csv")
df = pd.read_csv(MOBILE)

X = df.drop(columns="price_range")
y = df["price_range"]

from sklearn.feature_selection import chi2

statistic, p_value = chi2(X, y)
table = pd.DataFrame({"feature": X.columns, "chi2": statistic, "p": p_value})
table = table.sort_values("chi2", ascending=False)

print(table.head(8).to_string(index=False))
print()
print(table.tail(4).to_string(index=False))
  • ram scores 931267.5, about fifty times the next column. Memory size decides a phone's price band almost on its own.
  • Four columns clear 9000, and all four are physical quantities: memory, screen pixels and battery capacity.
  • The p values for those four print as 0.0. They are smaller than the smallest number a float can hold, which is not a bug but a sign the statistic is enormous.
  • The bottom of the table is near zero. three_g scores 0.33 at a p value of 0.95, so the counts are what you would expect if the column and the price band had nothing to do with each other.
  • The scale is not comparable across column types. ram runs to 4000 and blue is 0 or 1, and chi square grows with the size of the counts, so the ranking is more trustworthy than the gaps between the numbers.

The scores span six orders of magnitude, so a log scale is the only way to see them all at once.

ExampleTwenty scores on a log axis
import matplotlib.pyplot as plt

ordered = table.sort_values("chi2")

plt.figure(figsize=(7, 5.5))
plt.barh(range(len(ordered)), ordered["chi2"], color="#9370DB")
plt.yticks(range(len(ordered)), ordered["feature"])
plt.xscale("log")
plt.axvline(9000, color="#d64541", linestyle="--", label="the gap below the top four")
plt.xlabel("chi square statistic, log scale")
plt.title("Chi square against the price band, 20 columns")
plt.legend(loc="lower right")
plt.show()
print("largest / smallest:", round(ordered["chi2"].max() / ordered["chi2"].min()))
A horizontal bar chart of the chi square statistic for each of the 20 mobile price columns on a logarithmic axis, smallest at the bottom. The bars climb smoothly from three_g at 0.33 to ram at 931267. A dashed red line at 9000 separates the top four columns, ram, px_height, battery_power and px_width, from the rest.

Keeping the best k columns

SelectKBest wraps the same score and returns a transformer, which is what goes into a pipeline. k is the number of columns to keep.

ExampleTen columns in, ten columns out
from sklearn.feature_selection import chi2, SelectKBest

selector = SelectKBest(score_func=chi2, k=10).fit(X, y)
kept = list(X.columns[selector.get_support()])

print("kept:   ", kept)
print("dropped:", [c for c in X.columns if c not in kept])
print()
print("shape in: ", X.shape, " shape out:", selector.transform(X).shape)

Six of the ten dropped columns are yes-or-no flags: blue, dual_sim, four_g, three_g, touch_screen and wifi. The other four hold small numbers: clock_speed, m_dep, n_cores and pc. On this dataset the price band is decided by what the hardware measures, not by which features are present.

The non-negative rule, and what breaks it

Chi square compares observed counts with expected counts, and a count cannot be negative. scikit-learn enforces this. Scaling a frame with StandardScaler, which centres every column on zero, makes half the values negative and breaks the test.

ExampleStandardising first is the common way to break this
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import chi2

X_scaled = StandardScaler().fit_transform(X)
chi2(X_scaled, y)
  • The error names the rule exactly. There is no warning and no partial result.
  • MinMaxScaler is the scaler that works here, because it maps every column into 0 to 1 and leaves nothing negative.
  • Chi square needs no scaling at all when the columns are already counts, which is the usual case for it.

Chi square against mutual information

Chi squareMutual information
Target typeClasses onlyClasses or a continuous value
Feature valuesNon-negative onlyAnything
ReturnsA statistic and a p valueA score with no significance test
ReproducibleExactly; it has no random stepOnly with random_state=
Reads a quantity like ramAs counts, so a large column scores largeOn its information, whatever its scale
Best forCounts, flags and encoded categoriesMixed columns and odd relationships

Where chi square fits

  • Text classification. A bag of words frame is non-negative counts against a class label, which is exactly the shape chi square was made for.
  • One hot encoded categories. Each dummy is 0 or 1 and non-negative, so the test asks whether a label and a class occur together more often than chance.
  • Survey and questionnaire data. Counts per answer per group is the textbook contingency table.
Watch out. Chi square assumes each expected cell count is not tiny, which is the rule of thumb that every expected count should be at least five. On a rare category against a rare class that assumption fails and the p value becomes unreliable. Group rare labels before the test, the same way One hot encoding with many categories groups them before encoding.
Try it yourself
  • Change k to 4 and print what survives; compare it with the four columns whose p value printed as 0.0.
  • Run chi2 on MinMaxScaler().fit_transform(X) and compare the ranking with the unscaled one; the same four columns lead, in a different order, which is how much the raw size of a column moves the statistic.
  • Split the data first with train_test_split(..., random_state=0, stratify=y), score only the training half and check that ram still leads.
PreviousMutual information

Little by little, you're building something great.