Efficient method to append pandas rows into a list

Question

I have a pandas DataFrame with ~5m rows. I am looking for an efficient method to append / store each rows into a list.

import pandas as pd

df = pd.DataFrame({ 
                    'id': [0, 1, 2, 3],
                    'val': ['w','x','y','z'],
                    'pos': ['p1','p2','p3','p4']
                 })

# Using List comprehensions 

df_lst = []
[df_lst.append(rows) for rows in df.iterrows()]

Given the size of the DataFrame; I am looking for other methods that are more efficient at storing rows to a list. Is there a vectorized solution to this?

list(df.iterrows()) is equivalent. But... 1. why do you need this? dataframe is the efficient method to store data. 2. You are using list-comprehensions wrong, this should be df_lst = [rows for rows in df.iterrows()] — Yevhen Kuzmovych
– Yevhen Kuzmovych, Commented Jan 12, 2023 at 21:10
I wonder why storing dataframe of 5M in a list should be called "efficient" — RomanPerekhrest
– RomanPerekhrest, Commented Jan 12, 2023 at 21:12

BehRouz · Accepted Answer · 2023-01-12 21:28:00Z

2

I'd recommend .tolist() as others also mentioned in the comments. So I'll give an example of it.

df_lst = df.values.tolist()

in terms of efficiency, which I see some have mentioned why you want to do that, that would depend on the use case. of course df is more efficient on performing various tasks on the data and it seems to be redundant to convert it, but note that a list is more memory-efficient than a dataframe. So converting that is not unrational if you don't need the features of the df.

answered Jan 12, 2023 at 21:28

BehRouz

1,3941 gold badge8 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

flipSTAR · Accepted Answer · 2023-01-12 21:35:39Z

0

From here:

You can use df.to_dict('records') to convert the rows into a list of dicts. If this is useful depends on what you want to do with the list afterwards.

answered Jan 12, 2023 at 21:35

flipSTAR

6431 gold badge6 silver badges22 bronze badges

Collectives™ on Stack Overflow

Efficient method to append pandas rows into a list

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related