How to insert a column into a DataFrame without generating the SettingWithCopyWarning

Question

I would like to insert a column into an existing DataFrame. Ideally without copying existing data. Whatever I try, later assignments to the resulting DataFrame generate a SettingWithCopyWarning, if the inserted data contains null values.

import pandas as pd

df = pd.DataFrame(data={'a': [1]})
df = df.assign(b=pd.Series(data=pd.NaT, index=df.index))
df['a'].iloc[0] = 5

Replacing assign with either of

df['b'] = pd.Series(data=pd.NaT, index=df.index)
df.insert(column='b', loc=0, value=pd.NaT)

results in the same warning.

Strange enough, if the inserted value is not null (replacing pd.NaT with, e.g., 0) doesn't generate a warning. Is that a bug?

Brad Solomon · Accepted Answer · 2017-11-18 22:47:11Z

1

Your issue seems to be with df['a'].iloc[0] = 5, where you're using chained assignment. Try this instead:

df.at[0, 'a'] = 5
# Or: df.loc[0, 'a'] = 5, but `.at` is preferred when assigning scalar

edited Nov 18, 2017 at 22:47

answered Nov 18, 2017 at 22:41

Brad Solomon

41.2k39 gold badges167 silver badges260 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Konstantin Over a year ago

Strange enough, the warning is not generated if I replace the null value with a non-null value. That is why I assumed that this chained assignment should work fine. Also, I would like to use position-based selection for rows and label-based selection for columns, which does not work with at, loc, iat, or iloc, AFAIK. and the ix is deprecated.

MaxU - stand with Ukraine Over a year ago

@Konstantin, for mixed (position + label) indexing use: df.loc[df.index[position], 'column_name']

Brad Solomon Over a year ago

Yeah, regarding .ix and combining position/label-based indexing: it might seem inconvenient, but .ix was deprecated specifically for that reason. The developers wanted to avoid ambiguity where possible--for example, if your index was [1, 0] and you used .ix, should this refer to a position or label?

Konstantin Over a year ago

@MaxU: Yes, I was aware of that possibility but found it ugly. df[column_name].iloc[position] is much more intuitive. But seems that df.loc[df.index[position], 'column_name'] is the only way to avoid the warning.

Collectives™ on Stack Overflow

How to insert a column into a DataFrame without generating the SettingWithCopyWarning

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related