Python's regex module matches invalid unicode character not in string

Question

I'm getting a strange behaviour running this code:

regex.search(ur'([^\p{IsAlnum}\s\.\'\`\,\-])', u'\U0001f618')

This should match \U0001f618, which is the unicode representation of a kissing emoji. The result, however, is the following:

<regex.Match object; span=(0, 1), match=u'\ud83d'>

This doesn't make sense at all, because u'\ud83d' is not even a valid unicode character.

I expected this instead:

<regex.Match object; span=(0, 1), match=u'\U0001f618'>

What is happening here?

I'm running Python 2.7.13 on macOS Sierra 10.12.6, regex.__version__ is 2.4.130.

Cannot reproduce. Same python version, same regex version, output is <regex.Match object; span=(0, 1), match=u'\U0001f618'>. I'm on Manjaro instead of Mac, but not sure how that would make a difference. Maybe try reinstalling the regex module? — Aran-Fey
– Aran-Fey, Commented Oct 4, 2017 at 10:32
Same Python and regex versions, however, on Linux platform. Works as you expect. — mhawke
– mhawke, Commented Oct 4, 2017 at 10:35
Same Python and regex versions, can reproduce on macOS Sierra 10.12.6: <regex.Match object; span=(0, 1), match=u'\ud83d'> — jbndlr
– jbndlr, Commented Oct 4, 2017 at 10:41
This may help: Python returns length of 2 for single Unicode character string — PM 2Ring
– PM 2Ring, Commented Oct 4, 2017 at 11:13

user41951 · Accepted Answer · 2017-10-04 16:03:10Z

1

As mentioned by @PM 2Ring, it is happening because Python is compiled with UCS-2 support (narrow range) instead of UCS-4 support (wide range). Because of this, Python internally (and incorrectly) represents u'\U0001f618' as two characters, which explains the regex result.

More information here: https://stackoverflow.com/a/29109996/4111012

answered Oct 4, 2017 at 16:03

user41951

1911 gold badge1 silver badge10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python's regex module matches invalid unicode character not in string

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related