pandas read_csv Fails on Large String

Question

Python 3.7
pandas 0.24.2

Within a Python script, I am loading a zip of many CSVs from a remote server. I am then reading a subset of these CSVs to utf-8 strings, after which I attempt to read each CSV string into a dataframe using .read_csv(). Attempts to carry out this action are failing. For a single string:

large_string = '"num","domain_name",...,"clientTransferProhibited","","","";'

df = pd.read_csv(large_string, sep=',', engine='python')

Expected: return a dataframe and run rest of script.
Observed: when run from terminal, all of large_string gets printed to the console and then the script ends (i.e. does not complete other code, but no error messages). When run in PyCharm, same thing happens, but I also see that it is stalling out down the pandas stack:

pandas>io>parsers>common.py

on line 427: f = open(path_or_buf, mode, errors='replace', newline="")

In PyCharm debug, within the common.py frame, I am able to see that the exception Error 63 File name too long is being thrown (although for some reason this exception never gets printed to console).

What is going on here? Is read_csv misinterpreting my data as a filepath? How do I resolve this?

xibalba1 · Accepted Answer · 2019-05-12 18:07:15Z

2

read_csv was, in fact, misinterpreting the data as a file path.

The solution is to convert the string to a text buffer, which is itself passed to read_csv:

import io

large_string = '"num","domain_name",...,"clientTransferProhibited","","","";'

df = pd.read_csv(io.StringIO(large_string), sep=',', engine='python')

answered May 12, 2019 at 18:07

xibalba1

5448 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

taless · Accepted Answer · 2019-05-13 02:54:55Z

1

An easy solution is to use split:

 df = pd.DataFrame([x.split(',') for x in large_string.split('\n')])

answered May 13, 2019 at 2:54

taless

1661 silver badge5 bronze badges

1 Comment

xibalba1 Over a year ago

This partially works. The catch is that my data has a header row, and this puts the headers as the first row, which isn't such a big deal on its own, but this has the add'l effect of messing up the column type inferences done by pandas.

Collectives™ on Stack Overflow

pandas read_csv Fails on Large String

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related