Two errors fetching data from the web with Pandas (IncompleteRead & urlopen error)

Question

I have tried to fetch data from the web (a csv file) using Pandas in Jupyter Notebook:

import pandas as pd
df1 = pd.read_csv("https://www.crowdflower.com/wp-content/uploads/2016/03/gender-classifier-DFE-791531.csv")

The first time I get the following error:

IncompleteRead: IncompleteRead(5738795 bytes read, 2437944 more expected)

I try it again in a different cell in jupyter notebook and get another error:

URLError:

I try a third time and Jupyter Notebook keeps hanging for ages

Any idea what these two errors means (what is pandas trying to tell me, what happened), and how to fix them?

Other than the encoding issue, it works fine for me. (I used df1 = pd.read_csv("https://www.crowdflower.com/wp-content/uploads/2016/03/gender-classifier-DFE-791531.csv", encoding='latin')). — user2285236
– user2285236, Commented May 27, 2017 at 18:07

mattmc3 · Accepted Answer · 2017-05-27 18:29:51Z

1

If you use curl to download the file, or hit it with a web browser that shows the text, you'll see that the file is not UTF-8 encoded, which is what Pandas assumes it is. I cannot tell you what the encoding should be for this dataset, but you can cheat and use ISO-8859-1 to at least get it loaded and simulate the naive (and totally untrue) assumption that 1 byte == 1 char until you can get a handle on what the encoding should be.

import pandas as pd
url = "https://www.crowdflower.com/wp-content/uploads/2016/03/gender-classifier-DFE-791531.csv"
df1 = pd.read_csv(url, encoding="iso-8859-1")
print(df1)

Then, read up on this. It's an oldie, but a goodie: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) . Like he says, "No excuses!"

answered May 27, 2017 at 18:29

mattmc3

18.6k8 gold badges88 silver badges109 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Two errors fetching data from the web with Pandas (IncompleteRead & urlopen error)

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related