Ascii error when trying to write in file open with codecs lib

Question

Code

# -*- coding: ISO-8859-15 -*-
import sys
import codecs
filename2 = "log_unicode2.log"
log_file2 = codecs.open(filename2, "w", "utf-8")
sys.stdout = log_file2
log_file2.write('aééé')

Error

Traceback (most recent call last):
  File "snippet_problem_unicode.py", line 7, in <module>
    log_file2.write('a├®├®├®')
  File "C:\Users\dev1\Envs\atao\lib\codecs.py", line 691, in write
    return self.writer.write(data)
  File "C:\Users\dev1\Envs\atao\lib\codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal
not in range(128)

Contexte

Windows 7
Python 2.6
Eclipse

'aééé' is a byte string (latin-1) which need to be converted to utf-8. Why do this conversion involve ascii codec ?

Martijn Pieters · Accepted Answer · 2014-08-29 12:28:40Z

1

You are writing a byte string to a file object expecting a unicode value. To go from byte string to unicode value Python has to decode the bytestring. This decoding uses the default ASCII codec.

Either:

Use a unicode literal instead of a byte string:
```
log_file2.write(u'aééé')
```
Explicitly decode the bytestring to Unicode first, using your source file encoding:
```
log_file2.write('aééé'.decode('latin1'))
```

Not use codecs.open() but open the file using the built-in open() function instead, then manually decode, then encode to UTF:

log_file2 = open(filename2, "w")
log_file2 .write('aééé'.decode('latin1').encode('utf8')

or use a unicode literal and encode manually:

log_file2 = open(filename2, "w")
log_file2 .write(u'aééé'.encode('utf8'))

answered Aug 29, 2014 at 12:28

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

LBarret Over a year ago

Said diffrently : a byte string is implicitely the result of a former ascii encoding and is decoded as such by the codecs lib.

Martijn Pieters Over a year ago

@LBarret: a byte string may contain textual information. When attempting to use it as Unicode text it'll be decoded with the assumption it is using the ASCII codec.

Collectives™ on Stack Overflow

Ascii error when trying to write in file open with codecs lib

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related