Text with unicode CODES not displaying correctly in Python 3.7

Question

After being hours checking every stackoverflow post related to this, I'm pulling my hair to solve what should be a very simple thing...

I have a text file with unicode codes (not characters, codes), for example:

"Edward escribi\u00c3\u00b3 la biograf\u00c3\u00ada de su autor favorito"

This string should be displayed as:

"Edward escribió la biografía de su autor favorito"

If I load the file as I normally do, I will just see the unicode strings when I print the loaded text:

import io
chars = io.open(fb_json_path, encoding='utf-8').read().strip()

This prints: "Edward escribi\u00c3\u00b3 la biograf\u00c3\u00ada de su autor favorito". It's the same if I remove the encoding parameter.

I guess I need to tell Python to interprete the codes in the string and display them as utf-8 characters, but I don't get how to do that.

Thanks in advance!

Are the quotes in the file?

Serge Ballesta
– Serge Ballesta

2020-05-02 17:32:43 +00:00
Commented May 2, 2020 at 17:32 — Serge Ballesta
– Serge Ballesta, Commented May 2, 2020 at 17:32
@SergeBallesta no, the file does not contain those quotes.

Eduardo G.R.
– Eduardo G.R.

2020-05-02 17:41:44 +00:00
Commented May 2, 2020 at 17:41 — Eduardo G.R.
– Eduardo G.R., Commented May 2, 2020 at 17:41

Thierry Lathuille · Accepted Answer · 2020-05-02 17:43:38Z

2

We can do that in two steps:

First, we read the file with encoding='unicode_escape' to convert all of the \uxxxx.

Then, we convert this to utf-8 by encoding it transparently to a bytes object (with latin-1 codec) and convert it to text again, decoding as utf-8

with open('text.txt', encoding='unicode-escape') as f:
    text = f.read()
    print(text)
    #Edward escribiÃ³ la biografÃa de su autor favorito

    # Now we convert it to utf-8
    text = text.encode('latin1').decode('utf8')
    print(text)
    # Edward escribió la biografía de su autor favorito

answered May 2, 2020 at 17:43

Thierry Lathuille

24.4k10 gold badges49 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Eduardo G.R. Over a year ago

Beautiful !! Thanks so much @Thierry !

SubOptimal · Accepted Answer · 2020-05-02 17:42:04Z

0

I believe the input is somehow garbled by wrong encoding.

C3 93 are the UTF-8 encoding bytes for the ó(LATIN CAPITAL LETTER O WITH ACUTE).

run in Python 3 console

>>> text = "Edward escribi\u00c3\u00b3 la biograf\u00c3\u00ada de su autor favorito"
>>> text.encode('cp1252').decode('utf8')
'Edward escribió la biografía de su autor favorito'

answered May 2, 2020 at 17:42

SubOptimal

23.1k3 gold badges57 silver badges70 bronze badges

Collectives™ on Stack Overflow

Text with unicode CODES not displaying correctly in Python 3.7

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related