How do I get total row count of Pandas Dataframe when using iterator=True

Question

I'm dealing with large files that doesn't fit in memory, as a result of that I'm using the iterator functionality of Pandas' Dataframe and processing single chunk each time.

pd.read_csv(csv_file_name, encoding='utf-8', chunksize=chunk_size, iterator=True,
                                            engine='c', error_bad_lines=False, low_memory=False)

While processing I'd like to print the number of processed rows and the percentage of processed rows out of the total amount of rows.

To get the total amount of rows in a Pandas Dataframe I'm using

len(df.index)

But when trying to use it when using ierator I'm getting

AttributeError: 'TextFileReader' object has no attribute 'index'

Any way of doing that? (while not going over each chunk)

You won't know about bad lines until you process the chunk and so at best you're only going to get an estimate of the final total. If an estimate is good enough might as well just print the number of lines in the csv: see stackoverflow.com/q/41553467/2750819 if you need help with that. — Kent Shikama
– Kent Shikama, Commented Oct 28, 2019 at 10:56

Robert King · Accepted Answer · 2019-10-28 10:56:53Z

0

Two possible work arounds I would use:

Use the columns option and read the file in with just one column. It may be that is small enough you can read in one go, but if not iterate over that to count the number of rows.
Use the linux command wc -l to count the number of lines. If you have a header you need to remove one. e.g.

wc_output = subprocess.run(['wc','-l', 'csv_file_name'])
# wc_output.stdout will be of format ` N_lines filename`
# subtract 1 to remove header
n_rows = int(wc_output.stdout.split()[0]) - 1

answered Oct 28, 2019 at 10:56

Robert King

9946 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Robert King Over a year ago

Kent Shikama's comment has a link to a question with some better suggestions than mine :-) I have upvoted his comment.

Collectives™ on Stack Overflow

How do I get total row count of Pandas Dataframe when using iterator=True

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related