Pandas ParserError: Error tokenizing data. C error: EOF inside string

Question

I have data that is over 400,000 lines long. When running this code:

f=pd.read_csv(filename,error_bad_lines=False)

I get the following error:

pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at row 454751

My data by the end of the file looks like this:

BTC 9948    8718    1.57E+12    ASK
BTC 52      8718    1.57E+12    ASK
BTC 120     8718    1.57E+12    ASK
BTC 200     8718    1.57E+12    ASK
BTC 150     8718    1.57E+12    ASK
BTC 50      8718    1.57E+12    ASK
BTC 10      8718    1.57E+12    ASK
BTC 57      8718    1.57E+12    ASK
BTC 50      8718    1.57E+12    ASK
BTC 50191   8718

Line 454751 is this one: BTC 50 8718 1.57E+12 ASK
I tried running error_bad_lines=False as seen above but that still doesnt work. I also searched for quotes in my file but I do not have any.

wjandrea · Accepted Answer · 2024-03-23 15:17:25Z

28

Changing the Parser engine from C to Python should solve your problem. Use the following line to read your csv:

f = pd.read_csv(filename, error_bad_lines=False, engine="python")

From the read_csv documentation:

engine : {‘c’, ‘python’}, optional

Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.

edited Mar 23, 2024 at 15:17

wjandrea

34k10 gold badges69 silver badges105 bronze badges

answered Feb 21, 2020 at 20:15

Rahul P

2,7032 gold badges21 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

AspiringCoder Over a year ago

This works, thanks! However, it processes the data but shows this error after each run Skipping line 454752: unexpected end of data

Vikranth Over a year ago

Have you found the reason or the solution for this? @AspiringCoder

Roel Van de Paar Over a year ago

Generally speaking, the handy thing with [temporary] swapping to the Python parser is that the Python parser may give a better idea of the underlying error. After fixing this, it's potentially possible to swap back to the C parser.

wjandrea · Accepted Answer · 2024-03-23 15:18:54Z

0

please split and read file like below.

import pandas as pd

for chunk in pd.read_csv(filename, chunksize=<your_chunksize_here>)
    do_processing()
    train_algorithm()

edited Mar 23, 2024 at 15:18

wjandrea

34k10 gold badges69 silver badges105 bronze badges

answered Feb 21, 2020 at 20:09

Jacob Davidson

71 bronze badge

Comments

L Tyrone · Accepted Answer · 2024-07-16 23:55:36Z

0

In the new versions of Python, the argument on_bad_lines='False' is replaced by on_bad_lines='skip'. So try the below code:

f = pd.read_csv(filename, on_bad_lines='skip', engine='python')

It solved my error.

edited Jul 16, 2024 at 23:55

L Tyrone

8,36123 gold badges34 silver badges47 bronze badges

answered Jul 16, 2024 at 2:11

Rohan Bhoyar

11 bronze badge

Collectives™ on Stack Overflow

Pandas ParserError: Error tokenizing data. C error: EOF inside string

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related