Converting between charsets in python

Question

I need to output some strings (to stdout) and because windows console works in cp437, if the string contains any characters outside cp437, an exception is thrown.

I got around this by

encoding=sys.stdout.encoding
pathstr = path.encode(encoding,errors="replace").decode(encoding)
print(pathstr)

where path is the str i want to output. I'm fine with characters replaced by "?"

This doesn't seem good because it converts to a byte array and back to a str.

Is there a better way to achieve this?

I'm still new to python ( a week maybe ) and I'm using Win7 32 bit with cpython 3.3

bobince · Accepted Answer · 2013-04-29 01:20:35Z

3

This doesn't seem good because it converts to a byte array and back to a str.

If you want to write raw bytes to the stream, use .buffer:

pathbytes= path.encode(encoding, errors= 'replace')
sys.stdout.buffer.write(pathbytes)

...oh for the day that issue 1602 comes to something and we can avoid the Unicode horror of the Windows command prompt...

answered Apr 29, 2013 at 1:20

bobince

538k111 gold badges675 silver badges846 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:18:56Z

I'm fine with characters replaced by "?"

You could set PYTHONIOENCODING environment variable:

C:\> set PYTHONIOENCODING=cp437:replace

And print Unicode strings directly:

print(path)

In that case, if you are redirecting to a file; you could set PYTHONIOENCODING to utf-8 and get the correct complete output.

You could also try WriteConsoleW()-based solutions from the corresponding Python bug and see if they work on Python 3.3 e.g.:

import _win_console
_win_console.install_unicode_console()

print("cyrillic: цык.")

Where _win_console is from win_console.patch. You don't need to set the environment variable in this case and it should work with any codepage (with an appropriate console font, it might even show characters outside the current codepage).

All solutions for the problem of printing Unicode inside the Windows console have drawbacks (see the discussion and the reference links in the bug tracker for all the gory details).

mlissner · Accepted Answer · 2013-04-27 23:27:56Z

0

The best advice I ever heard about Unicode was to make a Unicode Sandwich:

Immediately convert any incoming text in your program into unicode.
Deal exclusively with Unicode in your program.
Export to whatever serialization format you want for your output.

In this case, you're basically doing just that. In a longer program, it would make sense to do this in the manner you describe, and I think you'd feel more comfortable about it.

The only change I'd make would be to encode to utf-8, then decode to cp437 on output.

answered Apr 27, 2013 at 23:27

mlissner

18.3k18 gold badges121 silver badges177 bronze badges

1 Comment

Cairnarvon Over a year ago

I can't make sense of your last sentence. You mean encode your unicode object to a UTF-8 string, and then decode that as if it were a CP437 string?

Collectives™ on Stack Overflow

Converting between charsets in python

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related