1

I'm having the following error while recursing the files in a directory and printing file names in the console:

Traceback (most recent call last):
  File "C:\Program Files\Python33\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
53: character maps to <undefined>

According to the error, one of the characters in the file name string is \u2013 which is an EN DASH – character different from the commonly seen - minus character.

I have checked my Windows encoding which is set to 437. Now, I see that I have two options to workaround this by either changing the encoding of Windows console or convert the characters in get from the file names to suit the console encoding. How would I go do that in Python 3.3?

Improve this question
0

2 Answers 2

Reset to default
5

Windows console is using cp437 encoding and there is a character \u2013 that isn't supported by that encoding. Try adding this to your code: 

sys.stdout = io.TextIOWrapper(sys.stdout.buffer,'cp437','backslashreplace')
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Is there a solution for Python 2?
@iamstudio, the issues is AttributeError: 'file' object has no attribute 'buffer' for sys.stdout.buffer on Python 2.
python -c "import sys;sys.stdout.buffer"
@techtonik Sorry! My bad. It may help you: gist.github.com/iamsudip/43e250d7b26f590c3d6c
@techtonik Read this too: macfreek.nl/memory/Encoding_of_Python_stdout
|
5

or convert the characters in get from the file names to suit the console encoding

Probably the console encoding is already correct (can't tell from the error message though). Code page 437 simply doesn't include that character so you won't be able to print it.

You can reopen stdout with a text encoder that has a fallback encoding, as demonstrated in iamsudip's answer which uses backslashreplace, to at least get readable (if not reliably recoverable) output instead of an error.

changing the encoding of Windows console

You can do this by executing the console command chcp 1252 before running Python, but that will still only give you a different limited repertoire of printable characters - including U+2013, but not many other Unicode characters.

In theory you can chcp to 65001 to get UTF-8 which would allow you to print any character. Unfortunately there are serious bugs in the C runtime's standard IO implementation, which usually make this unusable in practice.

This sorry state of affairs affects all applications that use the MS C runtime's stdio library calls, including Python and most other languages, with the result that Unicode on the Windows console just doesn't work in most cases.

If you really have to get Unicode out to the Windows console you can use the Win32 WriteConsoleW API directly using ctypes, but it's not much fun.

Improve this answer

5 Comments

Python 3.3 should be good with code page 65001, see docs.python.org/dev/whatsnew/3.3.html#codecs
@MarkRansom: It isn't in my experience. It seems the problem is for a console handle Windows internally routes WriteFile to WriteConsoleA, which returns the number of decoded Unicode characters instead of the number of bytes written. WriteFile happily returns this incorrect number back to the CRT's write function.
+1 @eryksun's description matches my experience of this bug. I found the same behaviour from the read operations too, which often results in a process stalled, waiting for further input that will never come. I'm not sure if it's the CRT or WriteFile to blame but whichever it is there is a workaround present to fix the behaviour for ANSI-default multibyte codepages like 932 and 936, but not for 65001. It is very sad that MS have still not fixed this known problem over many, many years.
ANSICON has a fix for the issue I described. It hooks WriteConsoleA to inject support for ANSI escape sequences. You can you can tell it to return the number of bytes instead of characters by setting the environment variable ANSICON_API=[!]program;....
It replaces WriteConsoleA with MyWriteConsoleA, where it will "[f]udge the (presumably) correct count". It also replaces WriteFile with MyWriteFile, which redirects to MyWriteConsoleA for a console handle.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.