Pandas: modify multiple dataframes (in a loop)

Question

I have multiple data frames that I want to do the same function for them. therefore I need to iterate over my frameworks.

# read text files 
df1 = pd.read_csv("df1.txt", sep="\t", error_bad_lines=False, index_col =None)
df2 = pd.read_csv("df2.txt", sep="\t", error_bad_lines=False, index_col =None)
df3 = pd.read_csv("df3.txt", sep="\t", error_bad_lines=False, index_col =None)

I have used the following code, however, it is not working (it means that all dataframes are still the same, and the changes do not affect them):

for df in [df1 , df2 , df3]:
    df = df[df["Time"]>= 600.0].reset_index(drop=True)
    df.head()

How I can iterate over them? and how can I overwrite dataframes?

It is because you modiying your dataframes only in your loop, but not overwriting the dataframes in your list. — Erfan
– Erfan, Commented Jun 2, 2020 at 7:20
_ it is not working:_ What does that mean? Have you done any debugging? Please provide a minimal reproducible example, as well as a clear description of the problem. See How to Ask, help center. — AMC
– AMC, Commented Jun 3, 2020 at 1:43

Roy2012 · Accepted Answer · 2020-06-05 05:47:25Z

5

The problem is that you're not changing the data frames in place, but rather creating new ones. Here's a piece of code that changes things in-place. I don't have your data, so I create fake data for the sake of this example:

df1 = pd.DataFrame(range(10))
df2 = pd.DataFrame(range(20))
df3 = pd.DataFrame(range(30))
df_list = [df1, df2, df3]

for df in df_list: 
    # use whatever condition you need in the following line
    # for example, df.drop(df[df["Time"] < 600].index, inplace=True)
    # in your case. 
    df.drop(df[df[0] % 2 == 0].index, inplace=True)
    df.reset_index(inplace = True)

print(df2) # for example

The result for df2 is:

   index   0
0      1   1
1      3   3
2      5   5
3      7   7
4      9   9
5     11  11
6     13  13
7     15  15
8     17  17
9     19  19

edited Jun 5, 2020 at 5:47

answered Jun 2, 2020 at 8:10

Roy2012

12.7k3 gold badges28 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sahel Over a year ago

Yes, I need to change the data frames in place, but how?. For "drop", you can use the "inplace". However, I want to write "df[df["Time"]>= 600.0]" , which doesnt have inplace method.

Roy2012 Over a year ago

It's a 'drop' of everything which is below 600, isn't it?

Roy2012 Over a year ago

Let me know if it answers your question.

sahel Over a year ago

yes it works for place in drop. But I need also to megre them later which does not have in place. However your solution for this exaple works, thank you

Shahir Ansari · Accepted Answer · 2020-06-05 05:55:11Z

3

If you just store the new df to another list or same list you are all good.

newdf_list = []                                 # create new list to store df
for df in [df1 , df2 , df3]:
    df = df[df["Time"]>= 600.0].reset_index(drop=True)
    df.head()
    newdf_list.append(df)                       # append changed df to new list

answered Jun 5, 2020 at 5:55

Shahir Ansari

1,86818 silver badges21 bronze badges

Comments

lsr729 · Accepted Answer · 2020-06-05 05:06:39Z

2

This might work:

df_list=[df1,df2,df3]

for df in range(len(df_list)):
    df=df_list[i]
    df_list[i]=df[df["Time"]>=600.0].reset_iundex(drop=True)

edited Jun 5, 2020 at 5:06

answered Jun 2, 2020 at 7:54

lsr729

8542 gold badges16 silver badges31 bronze badges

2 Comments

Roy2012 Over a year ago

This won't change the original data frames. Just the list.

sahel Over a year ago

it produces this error in "for" line:'int' object is not iterable

Collectives™ on Stack Overflow

Pandas: modify multiple dataframes (in a loop)

3 Answers 3

4 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related