mongodb insertion shows 'strings in documents must be valid UTF-8'

Question

this is my code

        for code, data in dict_data.items(): 

            try:
                collection2.insert({'_id':code,'data':data})

            except Exception as e:
                print code,'>>>>>>>', str(e)
                sys.exit()

it exited with

         524715 >>>>>>> strings in documents must be valid UTF-8

I could find out the error only by the try catch method. dict_data is a large dictionary which contains calculated values from other collection.

how can i fix this?

thanks

Are you using Python 2.x or Python 3.x ?

Roberto
– Roberto

2014-03-18 09:19:25 +00:00
Commented Mar 18, 2014 at 9:19 — Roberto
– Roberto, Commented Mar 18, 2014 at 9:19
I am using python 2.7.5

V Manikandan
– V Manikandan

2014-03-18 09:22:33 +00:00
Commented Mar 18, 2014 at 9:22 — V Manikandan
– V Manikandan, Commented Mar 18, 2014 at 9:22

Roberto · Accepted Answer · 2014-03-18 10:04:40Z

4

If you are using PyMongo and Python 2.x, you should use str in utf-8 or unicode strings. See: http://api.mongodb.org/python/current/tutorial.html#a-note-on-unicode-strings

If datais a dict with multiple strings you can convert all of them to unicode using following function:

def convert2unicode(mydict):
    for k, v in mydict.iteritems():
        if isinstance(v, str):
            mydict[k] = unicode(v, errors = 'replace')
        elif isinstance(v, dict):
            convert2unicode(v)

for code, data in dict_data.items(): 
    try:
        convert2unicode(data)
        collection2.insert({'_id':code,'data': data})
    except Exception as e:
        print code,'>>>>>>>', str(e)
        sys.exit()

Previous code will convert all str values in unicode, the "keys" keep untouched, depending on root cause you should also convert the "keys".

edited Mar 18, 2014 at 10:04

answered Mar 18, 2014 at 9:20

Roberto

9,2004 gold badges46 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

V Manikandan Over a year ago

Rec, data is a large dictionary in which many strings will be there. How is it possible to provide unicode for all those strings???

Roberto Over a year ago

I've updated my answer in order to use a helper function to replace all str values in the dict

Roberto Over a year ago

@ManikandanV, if the answer solved your question, please consider to mark it as "accepted".

Roberto Over a year ago

@SRC No, the unicode values are replaced in the input param mydict (data)

Collectives™ on Stack Overflow

mongodb insertion shows 'strings in documents must be valid UTF-8'

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related