python for loop with updated values

Question

I have a pandas dataframe with over 10k rows. I need to iterate through every row and do math based on the value of the previous row after it's updated. For loop is very slow.

Example DF:

for loop example:

for i in range(1,len(DF)):
  DF['b'] = DF['b'].[i-1]+DF['c']

I also tried

DF['b'] = DF['b'].shift(1)+DF['c']

but that won't use the updated value 'b'.

What is the best way to do such calculations?

ANSWERED: loc and iloc helps. The best way to do this is:

for i in range(1, len(DF)):
  DF.loc[i, 'b'] = DF.loc[i-1, 'b'] + DF.loc[i, 'c']

The particular math you wish to do is important. If it is simply addition, you could use cumsum. If it is more complicated, there may be no way to avoid looping over the rows. In that case, you may need Cython to boost performance. — unutbu
– unutbu, Commented Jun 22, 2015 at 20:23
@TigerhawkT3 UPDATED. I was translating an R script to Python. — WQF
– WQF, Commented Jun 22, 2015 at 20:24
You need to be specific. Given your description the obvious question is why not call cumsum on both columns? A minimal working example would be helpful. The devil is in the details... — unutbu
– unutbu, Commented Jun 22, 2015 at 20:45

wololo · Accepted Answer · 2015-06-22 22:14:18Z

1

Use iloc

for i in range(1,len(DF)):
    DF.iloc[i]['b'] = DF.iloc[i-1]['b']+DF.iloc['i']['c']

answered Jun 22, 2015 at 22:14

wololo

3452 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

WQF Over a year ago

This helped a little, but yields the SettingWithCopyWarning. I changed it to DF.loc[i, 'b'] = DF.loc[i-1, 'b']+DF.loc['i', 'c'] and it's much faster.

wololo Over a year ago

Yes, you can also turn off the SettingWithCopyWarning using following command pa.options.mode.chained_assignment = None # default='warn'

Collectives™ on Stack Overflow

python for loop with updated values

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related