Copy-on-write and chained assignment

Copy-on-write is the pandas rule that a write reached through an intermediate object lands on a temporary copy and is thrown away, so a chained call such as df['Age'].fillna(28, inplace=True) changes nothing and does not raise.

Last updated: 12 Oct, 2026 · pandas 3.0 · scikit-learn 1.9

The datasets this course uses loaded the tables. Before anything writes into one of them, this rule has to be known, because it is the one that costs most: the column looks filled, the notebook prints no error, and the model trains on the missing values you believed you had replaced. It became permanent in pandas 3.0.

Syntax: the two forms that write into a frame are an assignment back into the column, and .loc with the rows and the column chosen together.

df['col'] = df['col'].fillna(value)     # replace the column with the filled one
df.loc[row_mask, 'col'] = value         # rows and column in one indexing step
df.fillna({'col': value}, inplace=True) # in place on the frame, not on a column

Filling Age and changing nothing

Reading three columns of the Titanic table

usecols keeps the output small by reading only the three columns this lesson needs. isnull() gives a True or False per row and sum() counts the Trues.

import pandas as pd

TITANIC = ("https://raw.githubusercontent.com/krishnaik06/"
           "Feature-Engineering-Live-sessions/master/titanic.csv")
df = pd.read_csv(TITANIC, usecols=['Survived', 'Age', 'Fare'])
print(df['Age'].isnull().sum())   # 177 of the 891 ages are missing

The fill that older notebooks use

fillna replaces every missing value with the value passed to it, and inplace=True asks it to change the object it was called on rather than return a new one. The median of Age is 28.0.

# the call that older notebooks use, and that no longer writes into df
df['Age'].fillna(df['Age'].median(), inplace=True)

Counting the nulls on both sides of that line

warnings.catch_warnings(record=True) collects the warnings raised inside the block into a list instead of printing them, so the example can show which one pandas raised. Without it, the same warning appears as a red block in your notebook and the cell goes on running.

import warnings
import pandas as pd

TITANIC = ("https://raw.githubusercontent.com/krishnaik06/"
           "Feature-Engineering-Live-sessions/master/titanic.csv")
df = pd.read_csv(TITANIC, usecols=['Survived', 'Age', 'Fare'])
print("Age nulls before:", df['Age'].isnull().sum())

with warnings.catch_warnings(record=True) as caught:   # keep the warning instead of printing it
    warnings.simplefilter("always")
    df['Age'].fillna(df['Age'].median(), inplace=True)

print("Age nulls after :", df['Age'].isnull().sum())
print("pandas warned with:", caught[0].category.__name__)
print("the program carried on to this line")

Age nulls before: 177
Age nulls after : 177
pandas warned with: ChainedAssignmentError
the program carried on to this line

Why 177 nulls survived the fill

• 177 before and 177 after. The column was not touched. Every missing age is still missing.
• df['Age'] is an intermediate object. Selecting a column hands back a Series that behaves as a copy of it. fillna(..., inplace=True) fills that copy. The copy is then discarded, because nothing holds a reference to it.
• A ChainedAssignmentError was raised, and the next line still ran. The class is named like an exception and inherits from Warning, so Python prints it and carries on.
• Nothing downstream notices. The next cell computes a mean, trains a model or scales a column, over a table that still has holes in it.

Assigning the result back

Drop inplace=True and keep what fillna returns. The filled Series is then written into the frame by the assignment, which is an ordinary column write and not a chained one. The second form below calls fillna on the frame itself, with a dictionary of column to value, so there is no intermediate object at all.

df['Age'] = df['Age'].fillna(df['Age'].median())
print("assigned back      :", df['Age'].isnull().sum())

other = pd.read_csv(TITANIC, usecols=['Survived', 'Age', 'Fare'])
other.fillna({'Age': other['Age'].median()}, inplace=True)
print("frame level fillna :", other['Age'].isnull().sum())

assigned back      : 0
frame level fillna : 0

Both print 0. The rule to carry through the rest of the course: never call an in-place method on something you reached with df[...]. Assign the result back, or call the method on the frame.

Turning the warning into an error

A warning scrolls past. An exception stops the notebook at the line that is wrong. warnings.simplefilter("error", ChainedAssignmentError) promotes this one warning class to an exception, and it is worth putting at the top of every notebook in this subject.

import warnings
import pandas as pd
from pandas.errors import ChainedAssignmentError

warnings.simplefilter("error", ChainedAssignmentError)   # the warning becomes an exception

TITANIC = ("https://raw.githubusercontent.com/krishnaik06/"
           "Feature-Engineering-Live-sessions/master/titanic.csv")
df = pd.read_csv(TITANIC, usecols=['Survived', 'Age', 'Fare'])
df['Age'].fillna(df['Age'].median(), inplace=True)

Traceback (most recent call last):
  File "main.py", line 10, in <module>
    df['Age'].fillna(df['Age'].median(), inplace=True)
pandas.errors.ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment using an inplace method.
Such inplace method never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' instead, to perform the operation inplace on the original object, or try to avoid an inplace operation using 'df[col] = df[col].method(value)'.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html

The traceback points at the exact line, and the message is the one pandas was printing quietly a moment ago: the intermediate object always behaves as a copy, so use df.method({col: value}, inplace=True) or df[col] = df[col].method(value). With this filter set, a chained write cannot slip past you.

The same trap through a row mask

Filling is not the only chained write. Selecting rows and then assigning into the selection has the same shape and the same result: df[missing] builds a new frame, the assignment writes into that new frame, and it is discarded. .loc takes the rows and the column in one step, so there is nothing intermediate to write into.

import pandas as pd

TITANIC = ("https://raw.githubusercontent.com/krishnaik06/"
           "Feature-Engineering-Live-sessions/master/titanic.csv")
df = pd.read_csv(TITANIC, usecols=['Survived', 'Age', 'Fare'])
missing = df['Age'].isnull()

df[missing]['Fare'] = 0.0                  # selects rows, then writes into the selection
print("after df[mask]['Fare'] = 0 :", df.loc[missing, 'Fare'].max())

df.loc[missing, 'Fare'] = 0.0              # rows and column chosen in one step
print("after df.loc[mask, 'Fare'] =", df.loc[missing, 'Fare'].max())

after df[mask]['Fare'] = 0 : 227.525
after df.loc[mask, 'Fare'] = 0.0

• 227.525 after the chained write. The highest fare among the rows with a missing age is unchanged, so not one of those 177 rows was written to.
• 0.0 after the .loc write. Same rows, same column, same value, and this time it landed in df.
• df['Fare'][missing] = 0.0 fails the same way, for the same reason. The order of the two lookups makes no difference; what matters is that there are two of them.

Chained write vs single step write

Where this bites

• A helper that imputes a column. A function that creates an indicator column and then fills in place does half its job: the new column appears, the fill silently does not happen, and the return value looks right.
• Replacing category labels. df['Sex'].replace({'male': 1, 'female': 0}, inplace=True) leaves the text in place, and the encoder two cells later fails on a string.
• Capping outliers. df['Age'][df['Age'] > 60] = 60 leaves the maximum at 80.
• Working on a filtered slice. sub = df[df['Survived'] == 1] gives a new frame. Writing into sub changes sub and never reaches df, with no warning at all, because the two frames are separate under copy-on-write. Write through df.loc when you mean to change df.

Related

• Previous: The datasets this course uses
• Next: Text columns and the str dtype
• Reference: Copy-on-Write in the pandas user guide