How are these strings represented internally in Python interpreter ? I don't understand

Question

# -*- coding: utf-8 -*-

a = 'éáűőúöüó€'
print type(a)    # <type 'str'>
print a          # éáűőúöüó€
print ord(a[-1]) # 172

Why is this working ? Shouldn't be this a SyntaxError: Non-ASCII character '\xc3' in file ... ? There are unicode literals in the string.

When I prefix it with u, the results are different:

# -*- coding: utf-8 -*-

a = u'éáűőúöüó€'
print type(a)    # <type 'unicode'>
print a          # éáűőúöüó€
print ord(a[-1]) # 8364

Why? What is the difference between the internal representations in python ? How can I see it myself ? :)

Why should it be a syntax error to have bytes in a byte string? — Wooble
– Wooble, Commented Feb 12, 2013 at 18:15
The first is a str object containing the UTF-8 bytes that are in the file. The second is a unicode object formed by decoding the UTF-8. Use repr() to see the difference. — Wooble
– Wooble, Commented Feb 12, 2013 at 18:19
Why the downvotes? This seems like a legitimate question. To ask any clearer would require knowledge of the answer. — Jon-Eric
– Jon-Eric, Commented Feb 12, 2013 at 18:22

Martijn Pieters · Accepted Answer · 2013-02-12 18:25:26Z

11

There are unicode literals in the string

No, there are not. There are bytes in the string. Python simply goes with the bytes your editor saved to disk when you created the file.

When you prefixed the string with a u'', you signalled to python that you are creating a unicode object instead. Python now pays attention to the encoding you specified at the top of your source file, and it decodes the bytes in the source file to a unicode object based on the encoding you specified.

In both cases, your editor saved a series of bytes to a file, for the € character, the UTF-8 encoding is three bytes, represented in hexadecimal as E282AC. The last byte in the bytestring is thus AC, or 172 in decimal. Once you decode the last 3 bytes as UTF-8, they together become the Unicode codepoint U+20AC, which is 8364 in decimal.

You really should read up on Python and Unicode:

The Python Unicode HOWTO
Pragmatic Unicode by Ned Batchelder
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

edited Feb 12, 2013 at 18:25

answered Feb 12, 2013 at 18:19

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kissgyorgy Over a year ago

Wow, great answer thanks ! I already read the first link still didn't understand the difference. Now it's crystal clear ! :)

Collectives™ on Stack Overflow

How are these strings represented internally in Python interpreter ? I don't understand

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related