EDA and Feature Engineeringpandas 3.0 · scikit-learn 1.9 · seaborn 0.13 · Python 3.12+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
72 small wins to finish your pathNext lesson →

The first look at a dataset

The first look at a dataset is four calls in a fixed order, shape, head, info and describe, which answer how big the table is, what a row looks like, what type each column holds and how the numbers are spread.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

Reading a CSV that is not UTF-8 got the Zomato table into a frame called df, 9551 rows and 21 columns of restaurant data. Nothing has been looked at yet. These four calls are what to run before any chart and before any decision, and they are taught once here for the whole course.

Syntax: one attribute and three methods, all on the frame.

python
df.shape        # (rows, columns), an attribute, no brackets
df.head(n)      # the first n rows, 5 by default
df.info()       # one line per column: name, non-null count, dtype
df.describe()   # count, mean, std, min, quartiles and max, numeric columns only

Counting the rows and reading the first ones

Columns, dtypes and describe · from the EDA and Feature Engineering video · 4:50 to 8:02

shape is a tuple of rows and columns. columns lists the names, and head prints the first rows so that you can see what a value in each column looks like. to_string() keeps pandas from cutting the output short.

ExampleThe size of the table and its first three rows
print("rows and columns:", df.shape)
print("columns:", df.columns.tolist()[:6], "...")
print(df[['Restaurant Name', 'City', 'Cuisines', 'Aggregate rating']].head(3).to_string())
  • 9551 rows and 21 columns. Small enough to read into memory and work with directly.
  • Column names with spaces in them. 'Restaurant Name' can be reached with df['Restaurant Name'] and not with df.Restaurant Name. Square brackets work for every name, so prefer them.
  • Cuisines holds several values in one cell, separated by commas. A column that holds a list is a column that will need splitting before a model sees it.

Reading the dtype audit

info() prints one line per column with its name, how many values are not missing, and its dtype. It is the densest of the four calls: in one screen it gives the type audit and the missing value audit together.

ExampleOne line per column
df.info()
  • RangeIndex: 9551 entries, 0 to 9550 is the row index. Every row has a label, and here the labels are the numbers 0 to 9550.
  • Cuisines shows 9542 non-null while every other column shows 9551. That difference of 9 is the only missing data in the table.
  • int64 and float64 are whole numbers and decimals. Price range is stored as int64 although it is a rating from 1 to 4, which is a category wearing a number's clothes.
  • str is text. On pandas 2 these thirteen columns would say object instead.
  • memory usage: 1.5 MB is what the frame costs in RAM, worth a glance before loading a file a hundred times this size.

Reading the numeric summary

dtypes.value_counts() turns the type audit into a count per type, which is the first question in the EDA checklist. describe() then summarises the numeric columns: how many values, their mean and standard deviation, the smallest and largest, and the three quartiles. Three columns are shown here to keep the table readable.

ExampleThirteen text columns, eight numeric ones, and three of them summarised
print(df.dtypes.value_counts().to_string())
print()
print(df[['Average Cost for two', 'Aggregate rating', 'Votes']].describe().round(2).to_string())
  • 13 text columns and 8 numeric ones. That split decides everything downstream: the text columns need encoding, the numeric ones need scaling.
  • Average Cost for two has a mean of 1199 and a maximum of 800000. The 75th percentile is 700, so three quarters of the restaurants are under 700 and the largest value is more than a thousand times that. The mean sits far above the median of 400, which is what a long right tail does.
  • Aggregate rating has a minimum of 0 on a scale that starts at 1. A zero is not a bad rating here, it is a restaurant with no rating yet.
  • describe() skipped the thirteen text columns. It summarises numbers only. Pass include='str' to get counts and most frequent values for the rest.

What each of the four calls answers

CallThe question it answersWhat to look for
df.shapeHow big is thisRows you can hold in memory, columns you can read through
df.head()What does a row look likeCells holding several values, odd placeholders, an index that is a column in disguise
df.info()What type is each column, and what is missingA number stored as text, a category stored as a number, a non-null count below the row count
df.describe()How are the numbers spreadA mean far from the median, a minimum of 0 where 0 is impossible, a maximum far above the 75th percentile

Where the first look pays off

  • Before writing any other line. Four calls take ten seconds and decide which techniques the rest of the work needs.
  • After every merge or concat. The shape is how you check that a join added the columns you expected and no rows you did not.
  • When a number looks wrong. A mean that sits far from the median, or a maximum far above the 75th percentile, is the first sign of an outlier.
Watch out. describe() silently covers only the numeric columns, so a table with eight numeric columns out of twenty-one gives a summary of eight. Check the count of columns in the summary against df.shape[1] before concluding that the data looks fine.
Try it yourself
  • Run df.describe(include='str') and read the four rows it gives for the text columns: count, unique, top and freq.
  • Call df.tail(3) and compare it with head(3). Some files are sorted, and the last rows then look nothing like the first.
  • Run df.info(memory_usage='deep') and compare the number with the 1.5 MB the plain call reports.

You understood something today that you didn't yesterday.