Ascii codec cannot decode HTML when saving to file in python

Question

I have a text that is part of html. I would like to save it to a file.

This works fine in debug mode in Eclipse, but fails on runtime from shell. I am using a short example of html that fails.

xx = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
with codecs.open('myfile.htm'), 'wb', encoding="utf-8") as output:
    output.write(data)

and I get:

 Exception 'ascii' codec can't decode byte 0xe2 in position XXX: ordinal not in range(128)

where XXX is the position in the relevant file of the "strange" symbols, such as the EURO sign.

Why is this working from Eclipse and not from shell? How do I solve this?

I tried

HTMLParser.HTMLParser().unescape()
unquote()
unicode()

Nothing worked...

Is xx an actual variable in the code or just a fragment of the file you're giving as the example? — Paulo Bu
– Paulo Bu, Commented Apr 29, 2013 at 13:56
it's just a fragment, I zoomed in on it using the XXX location, since the original is a very big file — eran
– eran, Commented Apr 29, 2013 at 17:12
Haven't you solved yet the problem? The file you're trying to process, what's its encoding? — Paulo Bu
– Paulo Bu, Commented Apr 29, 2013 at 17:25
it's not a file, it's a result of a url call to a remote file. — eran
– eran, Commented Apr 29, 2013 at 17:30
It is in the connection header. If you have Firefox and Firebug for example, you can see it within the Net tab. There you have the get requests sent and the header where is defined the charset is Content-Type which for this page is Content-Type text/html; charset=utf-8 — Paulo Bu
– Paulo Bu, Commented Apr 30, 2013 at 12:39

Aya · Accepted Answer · 2013-04-29 14:00:55Z

1

The following code works for me...

# coding=utf-8

import codecs

data = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
with codecs.open('myfile.htm', 'wb', encoding="utf-8") as output:
    output.write(data.decode('utf-8'))

...but if the source data is already UTF-8 encoded, and you also want to write UTF-8 data, there's no need to decode it to a Python unicode object, then re-encode back to UTF-8. You can just do...

# coding=utf-8

data = '<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />'
with open('myfile.htm', 'wb') as output:
    output.write(data)

answered Apr 29, 2013 at 14:00

Aya

42.5k6 gold badges60 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Ascii codec cannot decode HTML when saving to file in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related