Understanding unicode and encoding in Python

Question

When I enter following in the python 2.7 console

>>>'áíóús'
'\xc3\xa1\xc3\xad\xc3\xb3\xc3\xbas'
>>>u'áíóús'
u'\xe1\xed\xf3\xfas'

I get the above output. What is the difference between the two? I understand the basics of unicode, and different kind of encoding like UTF8, UTF16 etc. But, I don't understand what is being printed on the console or how to make sense of it.

The output in python2.7 of 'áíóús' is '\xc3\xa1\xc3\xad\xc3\xb3\xc3\xbas'. Not sure where you get the dfasdf at the end. — solarc
– solarc, Commented Jan 18, 2019 at 20:03

wim · Accepted Answer · 2019-01-18 20:08:25Z

4

u'áíóús' is a string of text. What you see echoed in the REPL is the canonical representation of that object:

>>> print u'áíóús'
áíóús
>>> print repr(u'áíóús')
u'\xe1\xed\xf3\xfas'

The things like \xe1 are related to hexadecimal ordinals of each character:

>>> [hex(ord(c)) for c in u'áíóús']
['0xe1', '0xed', '0xf3', '0xfa', '0x73']

Only the last character was in the ascii range, i.e. ordinals in range(128), so only that last character "s" is plainly visible in Python 2.x:

>>> chr(0x73)
's'

'áíóús' is a string of bytes. What you see printed is an encoding of the same text characters, with your terminal emulator assuming the encoding:

>>> 'áíóús'
'\xc3\xa1\xc3\xad\xc3\xb3\xc3\xbas'
>>> u'áíóús'.encode('utf-8')
'\xc3\xa1\xc3\xad\xc3\xb3\xc3\xbas'

edited Jan 18, 2019 at 20:08

answered Jan 18, 2019 at 19:57

wim

368k114 gold badges682 silver badges818 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Understanding unicode and encoding in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related