Python - Django - Encoding UTF-8

Question

I have a problem with text encoding. The context: I'm working on a django 1.11, python 3.6 app (it started in python 2.7 and django < 1.11 and was upgraded later).
I have to use an API that doesn't recognize my text when there are characters with accents (é-è-^). The API works fine and accepts these characters when I use a script in python 3.9 (so it's not incriminating). The API also accepts a call from my app with text without accents. So the payload and headers are not incriminated either.

The api developer told me that he'd already had this kind of problem (never with Django python) and that it was due to my text not being encoded correctly in UTF-8.

I tried adding :

# coding: utf-8
from __future__ import unicode_literals

(in all the files that could see my message) without success

In the environment I've find:

LANG=C.UTF-8 (I tried en_US.UTF-8, without success)

I tried to add:

LC_CTYPE=UTF-8 (without success)

DEFAULT_CHARSET is not defined in the app, so it would be UTF-8 by default.

If you've experienced this or have any ideas, I'd love to hear from you. Thanks

Edit: Send class look like:

class MySMSAdapter:

self.message = 'One message with accent é not working'
self.recipient = ["+33600000000"]

def login():
    #PRIVATE CODE

def construct_messages(self) -> list:
    return [{'to': recipient, 'body': self.message } for recipient in self.recipients]

def get_payload(self) -> str:
    payload = f"""{{ "accountreference": '{self.account_reference}', 
            "messages": {self.construct_messages()},
            "from": '{self.sender_sms_senderId}',
            "characterset": "Unicode",
            }}
            """
    return payload

def get_send_headers(self) -> dict:
    return {
        "Authorization": f"Basic {self.session_key}",
        "Content-type": "application/json",
        "Accept": "application/json",
        "charset": "utf-8",
    }

def send(self) -> requests.Response:
    self.login()
    return requests.post(url=f"{BASEURL}v1.0/messagedispatcher", headers=self.headers, data=self.payload)

Try :

    def get_payload(self) -> str:
    payload = json.dumps({
        "accountreference": self.account_reference,
        "messages": self.construct_messages(),
        "from": self.sender_sms_senderId,
        "characterset": "Unicode"}
    )
    return payload

The send message is transformed into : 'One message with accent \xc3\xa9 not working' And the API decode this correctly Closed

unicode_literals deals with literals so (raw) string values in the code, not about handling unicode strings in the program. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Dec 20, 2024 at 8:00
How exactly are you sending what to that API? That's the point where you need to encode correctly. Without knowing the details here, there's not much we can help with. — deceze
– deceze ♦, Commented Dec 20, 2024 at 8:02
FWIW, don't cobble together JSON by hand! json.dumps a dict, or just pass a dict to requests.post(json=...) and it'll do the work for you. — deceze
– deceze ♦, Commented Dec 20, 2024 at 8:26
@deceze I received the message with the correct formatting, It's OK with me. Thanks — Barbadoug
– Barbadoug, Commented Dec 20, 2024 at 8:47

deceze · Accepted Answer · 2024-12-20 08:54:43Z

You're cobbling together JSON by hand, which is a bad idea, because it's error prone and values with special characters won't be correctly encoded/escaped. For this particular question, it also means you're passing a plain string to requests.post. requests will need to encode that to bytes to send it, and by default it'll encode it to Latin-1. If your other server is expecting UTF-8, that's where the problem is.

You should pass a dict to request.post(json=...), and requests will encode it to JSON for you, using UTF-8 as expected by the JSON spec:

payload = {'messages': self.construct_messages(), ...}
requests.post(json=payload, ...)

Alternatively, if you pass an already formed JSON string to requests.post(data=...), you must ensure the encoding is correct yourself:

payload = '{"messages": ...}'
requests.post(data=payload.encode('utf-8'), ...)

Collectives™ on Stack Overflow

Python - Django - Encoding UTF-8

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related