Python encoding issue (ascii > unicode)

Question

I am having a typical utf-8 encoding issue, but so far I haven't been able to resolve it.

class StatsTandE(webapp2.RequestHandler):
  def get(self):

    conn = rdbms.connect(instance=_INSTANCE_NAME, database='origami')
    cursor = conn.cursor()
    cursor.execute('SELECT ui.displayName, ui.title, ui.costCenter FROM views AS v')
    results = [[str(row[0]), str(row[1]), str(row[2])] for row in cursor.fetchall()]
    logging.info('results identified as %s', results)

    template_file_name = 'templates/stats_results.html'
    template_values = {
      'results': results,
    }

    path = os.path.join(os.path.dirname(__file__), template_file_name)
    self.response.out.write(template.render(path, template_values))
    conn.close()

The error I am seeing is:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 4: ordinal not in range(128)

Changing str(row[0]) to str(row[0]).encode('utf-8') didn't work...

Any suggestions?

Which version of python are you using ? Unicode issues are resolved in different way whether you are using python 2.X or python 3. — Adrien Plisson
– Adrien Plisson, Commented Jan 16, 2013 at 21:40
If the string is going to contain UTF-8 characters, you should use the unicode datatype in Python 2. — Borealid
– Borealid, Commented Jan 16, 2013 at 21:42
We are using Python 2.7. How would I set the unicode datatype? — kd1978
– kd1978, Commented Jan 16, 2013 at 21:49
what happens if you just remove all str calls? General rule for data that represents text: convert from bytes to Unicode as soon as possible on input, use Unicode everywhere inside your program, convert to bytes from Unicode as late as possible on output. — jfs
– jfs, Commented Jan 16, 2013 at 22:02

maddyblue · Accepted Answer · 2013-01-17 19:09:04Z

1

Try changing str(row[0]) to unicode(row[0]).

Update: As others have stated: you should not even be using unicode and str unless you need it. Try just row[0], row[1], row[2]. When you need to display it, do something like this: row[0].encode('utf8').

edited Jan 17, 2013 at 19:09

answered Jan 16, 2013 at 21:52

maddyblue

16.9k8 gold badges33 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kd1978 Over a year ago

That worked! However, how would I then convert the unicode once more into a string so that it displays nicely on the page? Would I decode it once more?

Collectives™ on Stack Overflow

Python encoding issue (ascii > unicode)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related