EDA and Feature Engineeringpandas 3.0 · scikit-learn 1.9 · seaborn 0.13 · Python 3.12+
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
72 small wins to finish your pathNext lesson →

Reading a CSV that is not UTF-8

An encoding is the rule that turns the bytes in a file into characters, and encoding='latin-1' is the argument that lets pandas.read_csv read a CSV whose bytes are not valid UTF-8.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

Text columns and the str dtype ended the part on the pandas traps. The work starts here, with a restaurant table that arrives inside a zip and refuses to open. Two errors stand between the file and a DataFrame, and both of them are common enough that meeting them here beats meeting them on your own data.

Syntax: encoding names the rule for decoding the bytes, and a file object from a zip is read exactly like a path.

python
pd.read_csv(path_or_url, encoding='latin-1')   # decode with latin-1 instead of UTF-8

with zipfile.ZipFile(...) as zf:          # a member of a zip
    with zf.open('inside.csv') as fh:     # opened as a file object
        pd.read_csv(fh, encoding='latin-1')

Opening a zip that holds more than one file

Reading the data set, and the encoding error · from the EDA and Feature Engineering video · 2:53 to 4:50

The video reads zomato.csv after unpacking the archive by hand. The code here opens the archive in memory instead, so there is no download and no unzip step before the read. The encoding problem that follows is the same one.

read_csv unpacks a zip by itself when there is exactly one file inside it. This archive holds eight, so pandas cannot guess which one to read and says so.

ExampleWhat a zip with eight members gives
import pandas as pd

ZIP = ("https://raw.githubusercontent.com/krishnaik06/"
       "5-Days-Live-EDA-and-Feature-Engineering/main/Zomatodataset.zip")
pd.read_csv(ZIP)

The error lists the members, which is already useful. To pick one of them, download the archive into memory and open it with the standard library.

Downloading the archive into memory

urllib.request.urlopen(ZIP).read() returns the bytes of the file. io.BytesIO wraps those bytes so they behave like an open file, and zipfile.ZipFile reads the archive out of it. Nothing is written to disk.

python
import io, zipfile, urllib.request
import pandas as pd

ZIP = ("https://raw.githubusercontent.com/krishnaik06/"
       "5-Days-Live-EDA-and-Feature-Engineering/main/Zomatodataset.zip")
zf = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(ZIP).read()))

Listing what is inside

namelist() gives the member names and getinfo(name).file_size gives the unpacked size of each, which is how you tell the table from the extras.

ExampleThe eight members of the archive
import io, zipfile, urllib.request
import pandas as pd

ZIP = ("https://raw.githubusercontent.com/krishnaik06/"
       "5-Days-Live-EDA-and-Feature-Engineering/main/Zomatodataset.zip")
zf = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(ZIP).read()))

for name in zf.namelist():
    print(f"{zf.getinfo(name).file_size:>9,} bytes  {name}")

Two members matter: zomato.csv is the table, and Country-Code.xlsx is a lookup that the merging lesson uses. The five JSON files hold the same restaurants in another format and are not needed.

The read that cannot decode

zf.open('zomato.csv') hands back a file object, and read_csv takes it like any other. The default encoding is UTF-8, and this file is not UTF-8.

ExampleThe default UTF-8 decode, on a file written in a legacy encoding
with zf.open('zomato.csv') as fh:
    pd.read_csv(fh)

The message is precise: at byte 7044 there is 0xed, and in UTF-8 that byte has to be followed by a particular pattern of continuation bytes, which it is not. Byte 7044 falls inside a city name, Brasília, written with a single byte for the accented letter. UTF-8 would use two.

Choosing an encoding

There is no way to read an encoding out of a plain text file: it is not recorded anywhere, so it has to be tried. Three are worth trying, in this order.

ExampleThree encodings against the same bytes
for enc in ['utf-8', 'cp1252', 'latin-1']:
    with zf.open('zomato.csv') as fh:
        try:
            frame = pd.read_csv(fh, encoding=enc)
            print(f"{enc:8} ok    {frame.shape}  row 22 city {frame.loc[22, 'City']!r}")
        except UnicodeDecodeError as err:
            print(f"{enc:8} fails {err}")

Reading the three attempts

  • utf-8 stops at byte 7044. It is the default and the right first guess, because most files written this decade are UTF-8.
  • cp1252 gets further and stops at byte 9144. It is the Windows western European encoding, and it leaves five of the 256 byte values undefined. 0x81 is one of them.
  • latin-1 reads the whole file and gives 9551 rows and 21 columns. It maps all 256 byte values to a character, so it never raises on any input.
  • Row 22 reads 'Brasí_lia', not 'Brasília'. latin-1 decoded the accented letter correctly and the stray underscore was in the file to begin with. An encoding that reads without error is not proof that every character came out right.

The frame the next lessons use

With the encoding settled, the read is one line. df is the Zomato table, and the rest of this part audits it.

Example9551 restaurants, 21 columns
with zf.open('zomato.csv') as fh:
    df = pd.read_csv(fh, encoding='latin-1')

print(df.shape)
print(df[['Restaurant Name', 'City', 'Cuisines']].head(3).to_string())

utf-8 vs cp1252 vs latin-1

utf-8cp1252latin-1
Also calledUTF-8, the defaultWindows-1252, ANSIISO-8859-1
Bytes per character1 to 411
Characters it coversEvery oneWestern European plus quotes and dashesWestern European
Can it fail?Yes, on an invalid byte sequenceYes, on 5 undefined byte valuesNo, every byte maps to something
On this fileStops at byte 7044Stops at byte 9144Reads all 9551 rows
Use it whenThe file is modern or you wrote itThe file came out of Excel on WindowsNo other encoding works and you can check the result

Where an encoding argument is needed

  • A file exported from an old system. Spreadsheets, banking extracts and survey tools often still write cp1252 or latin-1.
  • Data with names and places in it. A table of English words decodes as UTF-8 by accident, because the first 128 byte values mean the same thing in all three encodings. The first accented name is where it breaks.
  • A dataset downloaded from a competition or an archive. A zip of several files, one of which is the table, is the usual packaging.
Watch out. encoding='latin-1' never raises, which makes it a tempting default and a quiet way to corrupt text. Always look at a column of names afterwards. If accented characters come out as pairs of odd symbols, the file was UTF-8 after all and latin-1 read each byte on its own.
Try it yourself
  • Add 'utf-16' to the list of encodings and read the error it gives, which is a different one again.
  • Read the file with encoding='utf-8', encoding_errors='replace' and print frame.loc[22, 'City']. What has replaced the accented letter?
  • Open Country-Code.xlsx from the same archive with pd.read_excel and print its shape.
PreviousText columns and the str dtype

Little by little, you're building something great.