Django request with unicode strings wrongly UTF-8 encoded

Question

I'm having some trouble making my website compatible with accented characters (french website).

I have a form where some field values can be with accented chars: "Coupé" for instance.

My URL looks like this:

http://localhost:8080/recherches/s?marque=Audi&modeles=A5+Coup%C3%A9

In my django view I do something like this:

def search(request):
  logger = logging.getLogger('custom')
  criteria_form = CriteriaForm(request.GET or None)
  logger.debug("search")
  logger.debug(request.GET)

And what I get in my logs is:

<QueryDict: {u'marque': [u'Audi'], u'modeles': [u'A5 Coup\xc3\xa9']}>

If I query my database with this variable "modeles", I get an error:

>>> mo = u'A5 Coup\xc3\xa9'
>>> Vehicule.objects.filter(valid=True, modele=mo)[0].marque.name
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 211, in __getitem__
    return list(qs)[0]
IndexError: list index out of range

Things work if I query the database with the utf-8 version:

>>> mo = 'A5 Coup\xc3\xa9'
>>> Vehicule.objects.filter(valid=True, modele=mo)[0].marque.name
u'Audi'

So I think (but I might be wrong) that my problem comes from the fact that my variable is utf8 and then encoded with unicode.

How comes this is encoded that way?

UPDATE AFTER 1st RESPONSE:

On the header of the page that sends the form there is:

<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>

However if I print in my views.py the encoding:

logger.debug(request.encoding)

Then I get None.

But I don't know how to setup this encoding. I thought it would be from the header like I did above...

Also I have that in my HTTP_ACCEPT_CHARSET:

HTTP_ACCEPT_CHARSET ISO-8859-1,utf-8;q=0.7,*;q=0.3

Can that come from here? If yes, how should I change that?

I updated my post above. I thought it was encoding in UTF-8 but when I check request.encoding in my django views.py, I get None... I don't know how to set this encoding though :/ — EagleOne
– EagleOne, Commented Oct 28, 2012 at 18:01

EagleOne · Accepted Answer · 2012-10-30 07:26:25Z

I have finally found the problem. I tested this problem with a brand new django application with nearly nothing in it. No DB, a simple view which displayed the content of the form in a page. Something like this:

# -*- coding: utf-8 -*-
from django.shortcuts import render_to_response
from django.template import RequestContext


def test(request):
  found = request.GET.get('modeles')
  print found
  return render_to_response('test.html',
                            {"found":found},
                            context_instance=RequestContext(request))

And if I opened the url

localhost:8080/mysite?modeles=Coupé

I was getting the wrongly formatted CoupÃ©

Here for sure I could not blame any ajax, or db call or anything I could have done in python.

So I just tried it with Django 1.4... And it worked like a charm! Then I tried with the latest Django 1.5a1 version and it also worked...

I guess I should not use the beta for now. I will stick with the 1.4!

Hope this saves time for somebody else.

zxq9 · Accepted Answer · 2012-10-29 21:59:54Z

0

With all the validation magic happening in the background, it may be easier to pinpoint the problem by bypassing all that. In your view, what does the following produce?

Vehicule.objects.filter(valid=True, modele=request.GET['mo'])[0].marque.name

If all you are doing is passing a query in, then your huge Ajaxariffic validation set is going to be doing weird Python things to your unicode before you get the value back, and possibly knocking your special characters off the end.

EDIT: After playing with this myself, it looks like the urlified data within a GET is the problem. Instead of doing that and string.encode('iso8859-1').decode('utf8'), it is easiser to change your form to submit POST data instead and use that.

edited Oct 29, 2012 at 21:59

answered Oct 29, 2012 at 21:01

zxq9

13.2k1 gold badge48 silver badges63 bronze badges

2 Comments

EagleOne Over a year ago

If I do that I get the same problem. I get an error from the dB saying list index out of range. Because request.GET['mo'] is wrongly formatted. If I log it contains: A5 CoupÃ© ... Which is not in the db...

zxq9 Over a year ago

Edited after some further play.

Collectives™ on Stack Overflow

Django request with unicode strings wrongly UTF-8 encoded

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related