Default encoding of Python 3's open() under Windows. sys.getdefaultencoding() setting not respected

Question

I've been pulling my hair out over this issue for several hours now.

I have a message file I want to generate using django's makemessages command, which works just fine in a Linux environment which actually handles locale settings in a sane way. However, when I try the same thing under Windows, every time python tries to open a file, it assumes it is encoded in cp932 (SHIFT-JIS), which causes all sorts of havoc.

Manually adding encoding='utf-8' to every open call works, but that's hardly a good way of fixing the problem. Is there any way to force open to use a specific default encoding?

sys.getdefaultencoding() returns 'utf-8', for some arcane reason this setting is not respected
PYTHONIOENCODING and PYTHONENCODING are both set to 'utf-8'
My code page is set to cp65001

This is my python version string:

Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32

EDIT: I've noticed that locale.getpreferredencoding() returns cp932, so I guess finding a Windows locale with utf-8 as its default would do the trick. Does such a thing even exist?

It may be a bug in 3.3 on Windows if you really can't find a solution. On the other hand, being explicit about the encoding when calling open() is not a bad idea. — Simeon Visser
– Simeon Visser, Commented Dec 12, 2014 at 21:43
Well, I know that being explicit about the encoding is never a bad idea, the issue is that replacing all calls to open in the relevant parts of Django's utilities isn't really feasible. I've discovered that locale.getpreferredencoding() does return cp932, so that might be the issue. Are there any Windows locales that set this to utf-8? — mval
– mval, Commented Dec 12, 2014 at 21:46
Possibly, I'm not too familiar with that. But the fact that it returns cp932 may indeed indicate that it's the locale setting on your machine (or Windows in general) that interferes here. — Simeon Visser
– Simeon Visser, Commented Dec 12, 2014 at 22:07
I should add that this problem can (in principle) be solved by overriding locale.getpreferredencoding (which I temporarily did to get past the issue), but I really hope there's an alternative way to accomplish this. — mval
– mval, Commented Dec 12, 2014 at 22:07
You might suggest to Django developers that they let you, rather than Microsoft, specify the encoding that you want Django to use in its internal open calls. In the meanwhile, upgrade at least to 3.3.5 for its bugfixes; there might be some related to unicode. — Terry Jan Reedy
– Terry Jan Reedy, Commented Dec 13, 2014 at 19:51

Lazik · Accepted Answer · 2015-06-16 03:26:31Z

3

Try this

import locale
locale.setlocale(locale.LC_ALL, 'en_US.utf-8')

answered Jun 16, 2015 at 3:26

Lazik

2,5302 gold badges27 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Steven Over a year ago

This did not work for me.

Lazik Over a year ago

This is for python3 and it's just a way to avoid writing the encoding in all system access in a python script. For changing your windows shell encoding, try: chcp 65001

Collectives™ on Stack Overflow

Default encoding of Python 3's open() under Windows. sys.getdefaultencoding() setting not respected

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related