Pandas, build new dataframe with for loop

Question

I have a really simple problem that I cant solve in Pandas. I have a dataframe to start with, with that dataframe I want to apply some function. I want to repeat this many times and build/stack the reults from the operations in a new larger dataframe. I was thinking of doing this with a for loop. Here is a simplified example that I can not get to work:

import pandas as pd

df = pd.DataFrame(np.random.randn(3, 4), columns=list('ABCD'))

large_df = df*0

for i in range(1,10):
    df_new = df*i
    large_df= pd.concat(large_df,df_new)

large_df

Any ideas??

large_df = pd.concat([large_df,df_new])

behzad.nouri
– behzad.nouri

2014-03-21 11:15:17 +00:00
Commented Mar 21, 2014 at 11:15 — behzad.nouri
– behzad.nouri, Commented Mar 21, 2014 at 11:15
You could just append: large_df= large_df.append(df_new)

EdChum
– EdChum

2014-03-21 11:16:53 +00:00
Commented Mar 21, 2014 at 11:16 — EdChum
– EdChum, Commented Mar 21, 2014 at 11:16

Dan Allan · Accepted Answer · 2014-03-21 11:49:23Z

5

It will be fastest to build all of the results first and concatenate once in the end. If you append one result at a time, the memory for the results has to be re-allocated each time.

So, if you are applying some_function with a different parameter p through the loop (like i in your toy example above) I suggest:

pd.concat([df.apply(lambda x: some_function(x, p)) for p in parameters])

answered Mar 21, 2014 at 11:49

Dan Allan

35.5k6 gold badges72 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pandas, build new dataframe with for loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related