Processing CSV with Python

Question

*I am having an issue processing CSV'S. I get this error:

return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 22: character maps to <undefined>

What do I need to do to fix this? I think there is an issue where the CSV matches with the user's input.*

import csv

csvanswer=input("Type your issue here!").lower() 

searchanswer=(csvanswer).split() 

userinput = open("task2csv.csv")  

csv_task=csv.reader(userinput)

for row in csv_task:
    if row[0] in searchanswer: 

        print(row) 

        break

What character set is task2csv.csv using? You may have to use a codecs.open on it. — mpez0
– mpez0, Commented Mar 1, 2016 at 21:06
@mpez0: For Py3 code, you don't need (and shouldn't use) codecs.open; the open built-in supports multiple encodings already (and it's usually faster than codecs.open IIRC). Even on Py2.6/2.7, you usually want to use io.open, not codecs.open; it's only on 2.5 and earlier that codecs is the best option (by virtue of no other good options existing). — ShadowRanger
– ShadowRanger, Commented Mar 1, 2016 at 21:09
I have seen this error when my windows concole was not able to display characters my python script was outputting. As others have mentioned, make sure your .csv file does not have "smart quotes" and such. You can check by opening it in a text edior such as notepad++ and looking for weird characters. Also, try using pprint module instead of print. I think that allowed me to print. — Igor
– Igor, Commented Mar 1, 2016 at 21:32
@ShadowRanger Thanks for the info -- I was only considering 2.x, and hadn't noted io.open. I'm using codecs elsewhere in my code, and just used codecs.open. — mpez0
– mpez0, Commented Mar 2, 2016 at 14:36

ShadowRanger · Accepted Answer · 2016-03-01 21:08:01Z

1

Your input file is probably in an encoding other than the default on your system. You can fix this by explicitly providing the correct file encoding to the open call (you should also pass newline='' to the open call to properly obey the csv module's requirements).

For example, if the file is UTF-8, you'd do:

userinput = open("task2csv.csv", encoding='utf-8', newline='')

If it's some other encoding (UTF-16 is common for files produced by Windows programs), you'd use that. If it's some terrible non-UTF encoding, you're going to have to figure out the locale settings on the machine that produced it, they could be any number of different encodings.

answered Mar 1, 2016 at 21:08

ShadowRanger

158k12 gold badges221 silver badges316 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Bakerboy316 Over a year ago

Thanks for the reply, but that line of code didn't work

ShadowRanger Over a year ago

@Bakerboy316: Have you actually checked the encoding of the file? That line only works if the file is UTF-8, but it could be UTF-16, or any number of other weird encodings.

Mr Lister Over a year ago

@ShadowRanger UTF-16 doesn't usually contain bytes with value 0x90, unless it's a Chinese text.

Collectives™ on Stack Overflow

Processing CSV with Python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related