Pandas Python: Writing output to specific cells in the CSV

Question

I am trying to write a pandas df to a csv. However, I need to loop a lot to get my output in chunks. I want to stack the output vertically in a csv. So after each iteration, I want to write to a particular set of rows (1:10, then 11:20, etc...) and clear memory so as not to make a giant object. Is it possible to use df.to_csv to do this?

EdChum · Accepted Answer · 2014-08-21 14:01:36Z

5

to_csv accepts a mode argument to append:

import numpy as np
# df.shape returns the dimensions in a tuple, the first dimension is the number of rows
df_list = np.array_split(df, df.shape[0]/10)
for d in df_list:
    d.to_csv(csv_path, mode='a')

You can use numpy array split to produce a list of your df split into 10 rows each and then write them out or whatever you need to do to save memory

edited Aug 21, 2014 at 14:01

answered Aug 21, 2014 at 13:55

EdChum

397k204 gold badges837 silver badges583 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

FooBar Over a year ago

Oh wow, that's better. I just skimmed the docs and still missed it... sad day.

Collectives™ on Stack Overflow

Pandas Python: Writing output to specific cells in the CSV

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related