Difference between Python 3.3 and 3.4 `open` default encoding?

Question

I have a file with some non-ASCII characters.

$ file bi companies.txt
text/plain; charset=utf-8

On my desktop with Python 3.4 I can open this file with no problems:

>>> open('companies.txt').read()
'...'

On a CI system with Python 3.3 I get this:

>>> open('companies.txt').read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.3/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 1223: ordinal not in range(128)

But if I explicitly specify encoding='utf8', it works:

>>> open('companies.txt', encoding='utf8').read()
'...'

On both systems, sys.getdefaultencoding returns 'utf-8'.

Any ideas what is causing the systems to behave differently? Why is the CI system trying to use ascii?

Always specify the encoding if you know it, rather than relying on the default encoding, which may change. — kindall
– kindall, Commented Jan 14, 2015 at 0:33
That may be good advice, but my question is particular to why the two systems are behaving differently despite the default encoding (as far as I can tell) being the same. — Andrew Magee
– Andrew Magee, Commented Jan 14, 2015 at 0:40
@AndrewMagee. What does locale.getpreferredencoding() return on each system? — ekhumoro
– ekhumoro, Commented Jan 14, 2015 at 0:42
Ah, that looks like it could be it. On the CI system, that returns 'ANSI_X3.4-1968' so that would explain the difference. I wasn't aware of that being a thing. If you write that in an answer I will accept it. — Andrew Magee
– Andrew Magee, Commented Jan 14, 2015 at 0:49

ekhumoro · Accepted Answer · 2015-01-14 01:03:09Z

2

The encoding for text files is determined by locale.getpreferredencoding, rather than sys.getdefaultencoding.

answered Jan 14, 2015 at 1:03

ekhumoro

122k23 gold badges272 silver badges400 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Difference between Python 3.3 and 3.4 `open` default encoding?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related