Getting rid of SettingWithCopyWarning in Python pandas

Question

I am loading a bunch of csvs and processing certain columns if they exist, after loading the csv with pandas

data = pd.read_csv('Test.csv', encoding = "ISO-8859-1", index_col=0)

this dataframe will be used in the example

import pandas as pd
data = pd.DataFrame({'A': [1, 2.1, 0, 4.7, 5.6, 6.8],
                    'B': [0, 1, 0, 0, 0, 0],
                    'C': [0, 0, 0, 0, 0, 1],
                    'D': [5, 5, 6, 5, 5.6, 6.8],
                    'E': [2, 4, 1, 0, 0, 5],
                    'F': [0, 0, 1, 0, 0, 0],
                    'G': [0, 0, 0, 0, 0, 0],})

Next I check and select specific columns that are going be processed

coltitles = ['A', 'B','C', 'D', 'E']

columns = []

for name in coltitles:
    if name in data.columns:
        columns.append(name)
    else:
        print (name, 'is missing')

df = data[columns]

if 'A' in df.columns:
    #perform some processing, I will put print to simplify it
    print ('Exist')

The code works if I use a dataframe for data, but If I load the data from a csv I get a Warning:

<module3>:74: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

The warning is caused by the line where df = data[columns].

The code still works with the warning but how do I get rid of this warning without suppressing it?

Is this actually your code, is coltitles = ['A', 'B','C', 'D', 'E'] ? What is columns when the exception happens? — Andy Hayden
– Andy Hayden, Commented Apr 19, 2015 at 7:58
Hmm, I thought this error could only occur if you assign a value to a slice. That is, if you did something like data[columns] = something for example. @ccsv are you sure the error is on the line that says df = data[columns] -- in which case, could you please paste a full working snippet that reproduces the error? — S Anand
– S Anand, Commented Apr 19, 2015 at 12:44
@SAnand That is the full warning. It is not an error as it still works. — ccsv
– ccsv, Commented Apr 19, 2015 at 19:51

Alexander · Accepted Answer · 2015-04-20 05:46:45Z

2

The chained assignment warnings / exceptions are aiming to inform the user of a possibly invalid assignment. There may be false positives; situations where a chained assignment is inadvertantly reported.

The purpose of this warning is to flag to the user that the assignment is carried out on a copy of the DataFrame slice instead of the original Dataframe itself.

You generally want to use .loc (or .iloc, .at, etc.) type indexing instead of 'chained' indexing which has the potential to not always work as expected.

To make it clear you only want to assign a copy of the data (versus a view of the original slice) you can append .copy() to your request, e.g.

df = data[columns].copy()

See the documentation for more details.

answered Apr 20, 2015 at 5:46

Alexander

111k32 gold badges212 silver badges208 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Getting rid of SettingWithCopyWarning in Python pandas

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related