How to "normalize" python 3 unicode string

Question

I need to compare two strings. aa is extracted from a PDF file (using pdfminer/chardet) and bb is a keyboard input. How can I normalize first string to make a comparison?

>>> aa = "ā"
>>> bb = "ā"
>>> aa == bb
False
>>> 
>>> aa.encode('utf-8')
b'\xc4\x81'
>>> bb.encode('utf-8')
b'a\xcc\x84'

Mark Tolonen · Accepted Answer · 2020-05-18 22:01:21Z

11

You normalize with unicodedata.normalize:

>>> aa = b'\xc4\x81'.decode('utf8')   # composed form
>>> bb = b'a\xcc\x84'.decode('utf8')  # decomposed form
>>> aa
'ā'
>>> bb
'ā'
>>> aa == bb
False
>>> import unicodedata as ud
>>> aa == ud.normalize('NFC',bb)  # compare composed
True
>>> ud.normalize('NFD',aa) == bb  # compare decomposed
True

edited May 18, 2020 at 22:01

answered Nov 4, 2017 at 7:53

Mark Tolonen

181k26 gold badges182 silver badges278 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to "normalize" python 3 unicode string

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related