Save errors to a variable while reading csv file

Question

I’m trying to read an unknown large csv file with pandas. I came across some errors so I added the following arguments:

df = pd.read_csv(csv_file, engine="python", error_bad_lines=False, warn_bad_lines=True)

It is working good and skipping offending lines, and errors are prompted to the terminal correctly, such as:

Skipping line 31175: field larger than field limit (131072)

However, I’d like to save all errors to a variable instead of printing them. How can I do it?

Note that I have a big program here and can't change the output of all logs from file=sys.stdout to something else. I need a case specific solution.

Thanks!

So do you really need to save the errors to a variable? Or are you asking how to log the errors to anywhere you want rather than just printing? For the logging part, you can look into how to redirect stderr (not stdout) to a file. — Code-Apprentice
– Code-Apprentice, Commented Feb 2, 2022 at 20:00
I need to both save it to a variable (I later send the errors to the user via API) and to keep them on stdout for my own use. — Itayst
– Itayst, Commented Feb 2, 2022 at 20:03

eshirvana · Accepted Answer · 2022-02-03 18:17:16Z

1

use on_bad_lines capability instead (available in pandas 1.4+):

badlines_list = []
def badlines_collect (bad_line: list[str]) -> None:
        badlines_list.append(bad_line)
        return None

df = pd.read_csv(csv_file, engine="python",on_bad_lines=badlines_collect)

edited Feb 3, 2022 at 18:17

answered Feb 2, 2022 at 20:04

eshirvana

24.7k3 gold badges28 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Itayst Over a year ago

At first I couldn't use it due to deprecated pandas version. After upgrading the package I wrote exactly what you suggested, except for the : list[str]) - > list[str]: (which raised an 'type' object is not subscriptable error for me) and it didn't work... What is this part used for? Can I do it without this part?

eshirvana Over a year ago

@Itayst you need version 1.4 for that feature

Itayst Over a year ago

Yeah as I said I upgraded the package, the issue is not with pandas but the syntax of the function. I'm getting error on the (bad_line: list[str]) -> None: part. How can I fix it?

Collectives™ on Stack Overflow

Save errors to a variable while reading csv file

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related