python json custom decode for LONG

Question

I've run into problem when testing insert into mondgodb python object which was extracted from some json response of web service using default json decoder, json.load(....

I've got this error:

OverflowError: BSON can only handle up to 8-byte ints

Problem is caused by some values in python object which are decoded as long, and mongodb can't handle those.

How can I solve this, any ideas here?

Custom decoder comes to mind, so when some big number is to be decoded to long > instead string to be used there.

Other solution would be: to get python object with some possible long fields in it and to convert all those to string.

I haven't much experience with python and json. Any help would be appreciated

There is suggestion that storing it as float would work. , here is email thread supportin that. I just need this to work on any complex decoded json, all longs to be converted to float. How this coluld be done? — Sole
– Sole, Commented Apr 6, 2012 at 14:15

Sole · Accepted Answer · 2012-04-19 13:38:13Z

2

I've finally came to satisfying solution for problem:

`req = urllib2.Request(url)
opener = urllib2.build_opener()
file_json = opener.open(req)
json_response = json.load(file_json, parse_int=str)`

so, key point here is: parse_int=str which will decode every JSON int to string, so pymongo will swallow it with no problem.

Hope this helps someone.

answered Apr 19, 2012 at 13:38

Sole

612 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

luke14free · Accepted Answer · 2012-04-06 14:21:56Z

0

you could do something like this

>>> import json
>>> longhandler = lambda obj: str(obj) if isinstance(obj, long) else None
>>> data=['hello long!',100L]
>>> json.dumps(data,default=longhandler)
["hello long!","100"]

answered Apr 6, 2012 at 14:21

luke14free

2,5491 gold badge18 silver badges25 bronze badges

3 Comments

Sole Over a year ago

This is nice piece of code, thanks @luke14free. Iterestingly, json.dumps(encoded_json_with_long_in_it) gets also clean number, no L attached to it. I need python object itself to be checked for presence of long and those to be converted to stings.

luke14free Over a year ago

Can't understand sorry. Can you provide an example?

Sole Over a year ago

I've updated original question. So, what I need is to go through decoded json (possible complex structure) and to convert all long fields to string. Do you have idea how to achieve this?

Tyler Brock · Accepted Answer · 2012-04-06 15:37:40Z

0

The problem is that python is interpreting the number it loads from json as an int.

If you run the following you will be able to see the max integer value for your system.

>>> import sys
>>> sys.maxint
9223372036854775807

This is 2^(64-1) so you need to figure out when the number python considers an int should be stored as a 32-bit int or a long and force it to be a long.

Just do something like this in your json decoder:

if abs(int(num)) > 2**31:
    long(num)

Then when you save it, it will be a long:

>>> from pymongo import Connection
>>> c = Connection()
>>> collection = c['test']['test']
>>> doc = {"longnum": long(123)}
>>> doc
{'longnum': 123L}
>>> collection.insert(doc)
ObjectId('4f7f0491add3616704000000')
>>> collection.find_one()
{u'_id': ObjectId('4f7f0491add3616704000000'), u'longnum': 123L}
>>>

answered Apr 6, 2012 at 15:37

Tyler Brock

30.3k16 gold badges82 silver badges82 bronze badges

4 Comments

Sole Over a year ago

Thanks @Tyler for your solution. You proposed that: "force it to be a long." > I want to exactly avoid this. I use python.json decoder, and it works propertly, it decodes big series of nums to long. I want to avoid that either extending json decoder and insert this: if abs(int(num)) > 2**31: str(num) or, I can use standard decoder and than go through object and cast all longs to strings. I don't know how to tackle either of those.

Tyler Brock Over a year ago

I'm confused now, in your original question you stated MongoDB doesn't handle longs. It does, they just need to actually be longs, as ints are 32 bit only. If you really want them to be strings, just use the code you suggested in the comment above. Basically you need to extend the JSONEncoder class and implement the default() method as described here: docs.python.org/library/json.html#json.JSONDecoder

Sole Over a year ago

please see latest comment I've added that solved problem for me. Thanks for help.

JJC Over a year ago

Note also that in Python3, sys.maxint is gone. The new sys.maxsize is similar, and, on systems with 8-byte integers, may be equivalent for the purposes here (i.e., to determine numbers too big to fit in a mongo 8-byte integer).

Collectives™ on Stack Overflow

python json custom decode for LONG

3 Answers 3

Comments

3 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related