UTF-8 coding in Python

Question

I have an UTF-8 character encoded with `_' in between, e.g., '_ea_b4_80'. I'm trying to convert it into UTF-8 character using replace method, but I can't get the correct encoding.

This is a code example:

import sys
reload(sys)  
sys.setdefaultencoding('utf8')

r = '_ea_b4_80'
r2 = '\xea\xb4\x80'

r = r.replace('_', '\\x')
print r
print r.encode("utf-8")
print r2

In this example, r is not the same as r2; this is an output.

\xea\xb4\x80
\xea\xb4\x80
관  <-- correctly shown

What might be wrong?

ShadowRanger · Accepted Answer · 2016-02-03 01:47:39Z

4

\x is only meaningful in string literals, you're can't use replace to add it.

To get your desired result, convert to bytes, then decode:

import binascii

r = '_ea_b4_80'

rhexonly = r.replace('_', '')          # Returns 'eab480'
rbytes = binascii.unhexlify(rhexonly)  # Returns b'\xea\xb4\x80'
rtext = rbytes.decode('utf-8')         # Returns '관' (unicode if Py2, str Py3)
print(rtext)

which should get you 관 as you desire.

If you're using modern Py3, you can avoid the import (assuming r is in fact a str; bytes.fromhex, unlike binascii.hexlify, only take str inputs, not bytes inputs) using the bytes.fromhex class method in place of binascii.unhexlify:

rbytes = bytes.fromhex(rhexonly)  # Returns b'\xea\xb4\x80'

edited Feb 3, 2016 at 1:47

answered Feb 3, 2016 at 1:41

ShadowRanger

158k12 gold badges221 silver badges315 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

UTF-8 coding in Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related