POSIX locale, UnicodeDecodeError

Question

Open a docker instance (e.g., docker run -ti ubuntu:16.04), create the Python files a.py

# -*- coding: utf-8 -*-
a = 'ö'

and r.py

with open('a.py') as f: exec(f.read())

When executing python r.py, all is file. When using python3, however, one gets the dreaded

Traceback (most recent call last):
  File "r.py", line 2, in <module>
    exec(f.read())
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31: ordinal not in range(128)

Curiously, I can reproduce this only on docker.

Any hints on what may be going wrong, and how to fix it? (Note that I can only modify r.py.)

Nico Schlömer · Accepted Answer · 2017-03-24 17:37:25Z

2

Starting docker with a UTF-8 locale makes it all work as expected

docker run -e LANG=C.UTF-8 -ti ubuntu:16.04

answered Mar 24, 2017 at 17:37

Nico Schlömer

59.7k36 gold badges216 silver badges291 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

anthony sottile · Accepted Answer · 2017-03-24 22:13:14Z

1

The difference between inside docker and outside of docker is likely the LANG environment variable -- many many base images default to no LANG set (or LANG=C, etc.). Setting an explicit LANG=en_US.UTF-8 will cause open to properly decode the source (in python 3)). If you want the python 3 behavior in python2, you can import io and use io.open.

exec in python2 / python3 also allows bytes.

One way to avoid the encoding problem is to read the source from disk as bytes:

with open('a.py', 'rb') as f:
    exec(f.read())

answered Mar 24, 2017 at 22:13

anthony sottile

72.1k19 gold badges199 silver badges248 bronze badges

1 Comment

Nico Schlömer Over a year ago

Thanks! I noticed that with python3 and a = {}; with open('a.py', 'rb') as f: exec(f.read(), a) (to store a.py's contents in a dict), this gives the error ERROR - failed to write data to stream: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ANSI_X3.4-1968'>. Fixed with setting the environment variable export PYTHONIOENCODING=UTF-8.

anthony sottile · Accepted Answer · 2017-03-24 22:08:55Z

0

Have you tried io?

with io.open('a.py', encoding='utf-8') as f:
    do stuff

edited Mar 24, 2017 at 22:08

anthony sottile

72.1k19 gold badges199 silver badges248 bronze badges

answered Mar 24, 2017 at 17:14

gold_cy

14.2k4 gold badges27 silver badges55 bronze badges

1 Comment

Nico Schlömer Over a year ago

Works with python3, fails with python: SyntaxError: encoding declaration in Unicode string.

Collectives™ on Stack Overflow

POSIX locale, UnicodeDecodeError

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related