When selecting data from a Pandas dataframe, sometimes a view is returned and sometimes a copy is returned. While there is a logic behind this, is there a way to force Pandas to explicitly return a view or a copy?
-
Can you provide a sample of the difference between a view and a copy?cwharland– cwharland2014-05-06 04:56:46 +00:00Commented May 6, 2014 at 4:56
-
See here for the rules: stackoverflow.com/questions/23296282/…Karl D.– Karl D.2014-05-06 04:59:09 +00:00Commented May 6, 2014 at 4:59
-
@cwharland, my understanding is modifications to a view also modify the primary dataframe (so a reference), and a copy is... a copy.calben– calben2014-05-06 05:03:00 +00:00Commented May 6, 2014 at 5:03
-
@KarlD., is that link the only way to handle the difference between views and copies?calben– calben2014-05-06 05:03:48 +00:00Commented May 6, 2014 at 5:03
-
1I'm not sure you can as this is due to numpy and not pandas, the docs show the various situations that should be avoided and due to the non-deterministic nature of the type of calls it is advised that chained assignment should be avoid and hence the warning, as I understand it only chained assignment is the access method that should be avoidedEdChum– EdChum2014-05-06 07:26:58 +00:00Commented May 6, 2014 at 7:26
1 Answer
There are two parts to your question: (1) how to make a view (see bottom of this answer), and (2) how to make a copy.
I'll demonstrate with some example data:
import pandas as pd
df = pd.DataFrame([[1,2,3],[4,5,6],[None,10,20],[7,8,9]], columns=['x','y','z'])
# which looks like this:
x y z
0 1 2 3
1 4 5 6
2 NaN 10 20
3 7 8 9
How to make a copy: One option is to explicitly copy your DataFrame after whatever operations you perform. For instance, lets say we are selecting rows that do not have NaN:
df2 = df[~df['x'].isnull()]
df2 = df2.copy()
Then, if you modify values in df2 you will find that the modifications do not propagate back to the original data (df), and that Pandas does not warn that "A value is trying to be set on a copy of a slice from a DataFrame"
df2['x'] *= 100
# original data unchanged
print(df)
x y z
0 1 2 3
1 4 5 6
2 NaN 10 20
3 7 8 9
# modified data
print(df2)
x y z
0 100 2 3
1 400 5 6
3 700 8 9
Note: you may take a performance hit by explicitly making a copy.
How to ignore warnings: Alternatively, in some cases you might not care whether a view or copy is returned, because your intention is to permanently modify the data and never go back to the original data. In this case, you can suppress the warning and go merrily on your way (just don't forget that you've turned it off, and that the original data may or may not be modified by your code, because df2 may or may not be a copy):
pd.options.mode.chained_assignment = None # default='warn'
For more information, see the answers at How to deal with SettingWithCopyWarning in Pandas?
How to make a view: Pandas will implicitly make views wherever and whenever possible. The key to this is to use the df.loc[row_indexer,col_indexer] method. For example, to multiply the values of column y by 100 for only the rows where column x is not null, we would write:
mask = ~df['x'].isnull()
df.loc[mask, 'y'] *= 100
# original data has changed
print(df)
x y z
0 1.0 200 3
1 4.0 500 6
2 NaN 10 20
3 7.0 800 9