86

I've been searching for quite a while with no success. My project isn't using Django, is there a simple way to serialize App Engine models (google.appengine.ext.db.Model) into JSON or do I need to write my own serializer?

Model:

class Photo(db.Model):
    filename = db.StringProperty()
    title = db.StringProperty()
    description = db.StringProperty(multiline=True)
    date_taken = db.DateTimeProperty()
    date_uploaded = db.DateTimeProperty(auto_now_add=True)
    album = db.ReferenceProperty(Album, collection_name='photo')
0

14 Answers 14

62

A simple recursive function can be used to convert an entity (and any referents) to a nested dictionary that can be passed to simplejson:

import datetime
import time

SIMPLE_TYPES = (int, long, float, bool, dict, basestring, list)

def to_dict(model):
    output = {}

    for key, prop in model.properties().iteritems():
        value = getattr(model, key)

        if value is None or isinstance(value, SIMPLE_TYPES):
            output[key] = value
        elif isinstance(value, datetime.date):
            # Convert date/datetime to MILLISECONDS-since-epoch (JS "new Date()").
            ms = time.mktime(value.utctimetuple()) * 1000
            ms += getattr(value, 'microseconds', 0) / 1000
            output[key] = int(ms)
        elif isinstance(value, db.GeoPt):
            output[key] = {'lat': value.lat, 'lon': value.lon}
        elif isinstance(value, db.Model):
            output[key] = to_dict(value)
        else:
            raise ValueError('cannot encode ' + repr(prop))

    return output
Sign up to request clarification or add additional context in comments.

6 Comments

There is a small mistake in the code: Where you have "output[key] = to_dict(model)" it should be: "output[key] = to_dict(value)". Besides that it's perfect. Thanks!
This code will fail when it encounters a UserProperty. I worked around it with doing "output[key] = str(value)" in the final else, instead of raising an error.
Great stuff. Small improvement is to use iterkeys() instead since you don't use "prop" there.
I've not tried all the possible types (date, GeoPt, ...), but it seems the datastore has exactly this method, and it's been working for strings and integers for me so far: developers.google.com/appengine/docs/python/datastore/… So I'm not sure you need to reinvent the wheel to serialize to json: json.dumps(db.to_dict(Photo))
@gentimouton That method is a new addition. It certainly didn't exist in 2009
|
61

This is the simplest solution I found. It requires only 3 lines of code.

Simply add a method to your model to return a dictionary:

class DictModel(db.Model):
    def to_dict(self):
       return dict([(p, unicode(getattr(self, p))) for p in self.properties()])

SimpleJSON now works properly:

class Photo(DictModel):
   filename = db.StringProperty()
   title = db.StringProperty()
   description = db.StringProperty(multiline=True)
   date_taken = db.DateTimeProperty()
   date_uploaded = db.DateTimeProperty(auto_now_add=True)
   album = db.ReferenceProperty(Album, collection_name='photo')

from django.utils import simplejson
from google.appengine.ext import webapp

class PhotoHandler(webapp.RequestHandler):
   def get(self):
      photos = Photo.all()
      self.response.out.write(simplejson.dumps([p.to_dict() for p in photos]))

5 Comments

hey thanks for the tip. this works great except I can't seem to serialize the date field. I get: TypeError: datetime.datetime(2010, 5, 1, 9, 25, 22, 891937) is not JSON serializable
Hi, thanks for pointing the issue. The solution is to convert the date object to a string. For instance you can wrap the call to "getattr(self, p)" with "unicode()". I edited code to reflect this.
To remove db.Model's meta fields, use this: dict([(p, unicode(getattr(self, p))) for p in self.properties() if not p.startswith("_")])
for ndb, see fredva's answer.
self.properties() didn't work for me. I used self._properties. Full line: return dict([(p, unicode(getattr(self, p))) for p in self._properties])
15

In the latest (1.5.2) release of the App Engine SDK, a to_dict() function that converts model instances to dictionaries was introduced in db.py. See the release notes.

There is no reference to this function in the documentation as of yet, but I have tried it myself and it works as expected.

3 Comments

I wonder if this has been removed? I get AttributeError: 'module' object has no attribute 'to_dict' when I from google.appengine.ext import db and use simplejson.dumps(db.to_dict(r)) (where r is an instance of a db.Model subclass). I don't see "to_dict" in google_appengine/google/appengine/ext/db/*
it has to be used like "db.to_dict(ObjectOfClassModel)"
for a ndb object, self.to_dict() does the job. If you want to make the class serializable by the standard json module, add 'def default(self, o): return o.to_dict()` to the class
7

To serialize models, add a custom json encoder as in the following python:

import datetime
from google.appengine.api import users
from google.appengine.ext import db
from django.utils import simplejson

class jsonEncoder(simplejson.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return obj.isoformat()

        elif isinstance(obj, db.Model):
            return dict((p, getattr(obj, p)) 
                        for p in obj.properties())

        elif isinstance(obj, users.User):
            return obj.email()

        else:
            return simplejson.JSONEncoder.default(self, obj)


# use the encoder as: 
simplejson.dumps(model, cls=jsonEncoder)

This will encode:

  • a date as as isoformat string (per this suggestion),
  • a model as a dict of its properties,
  • a user as his email.

To decode the date you can use this javascript:

function decodeJsonDate(s){
  return new Date( s.slice(0,19).replace('T',' ') + ' GMT' );
} // Note that this function truncates milliseconds.

Note: Thanks to user pydave who edited this code to make it more readable. I had originally had used python's if/else expressions to express jsonEncoder in fewer lines as follows: (I've added some comments and used google.appengine.ext.db.to_dict, to make it clearer than the original.)

class jsonEncoder(simplejson.JSONEncoder):
  def default(self, obj):
    isa=lambda x: isinstance(obj, x) # isa(<type>)==True if obj is of type <type>
    return obj.isoformat() if isa(datetime.datetime) else \
           db.to_dict(obj) if isa(db.Model) else \
           obj.email()     if isa(users.User) else \
           simplejson.JSONEncoder.default(self, obj)

Comments

4

You don't need to write your own "parser" (a parser would presumably turn JSON into a Python object), but you can still serialize your Python object yourself.

Using simplejson:

import simplejson as json
serialized = json.dumps({
    'filename': self.filename,
    'title': self.title,
    'date_taken': date_taken.isoformat(),
    # etc.
})

2 Comments

Yes, but I don't want to have to do this for every model. I'm trying to find a scalable approach.
oh and i'm really surprised that I can't find any best practices on this. I thought app engine model + rpc + json was a given...
4

For simple cases, I like the approach advocated here at the end of the article:

  # after obtaining a list of entities in some way, e.g.:
  user = users.get_current_user().email().lower();
  col = models.Entity.gql('WHERE user=:1',user).fetch(300, 0)

  # ...you can make a json serialization of name/key pairs as follows:
  json = simplejson.dumps(col, default=lambda o: {o.name :str(o.key())})

The article also contains, at the other end of the spectrum, a complex serializer class that enriches django's (and does require _meta -- not sure why you're getting errors about _meta missing, perhaps the bug described here) with the ability to serialize computed properties / methods. Most of the time you serialization needs lay somewhere in between, and for those an introspective approach such as @David Wilson's may be preferable.

Comments

3

Even if you are not using django as a framework, those libraries are still available for you to use.

from django.core import serializers
data = serializers.serialize("xml", Photo.objects.all())

1 Comment

Did you mean serializers.serialize("json", ...)? That throws "AttributeError: 'Photo' object has no attribute '_meta'". FYI - serializers.serialize("xml", Photo.objects.all()) throws "AttributeError: type object 'Photo' has no attribute 'objects'". serializers.serialize("xml", Photo.all()) throws "SerializationError: Non-model object (<class 'model.Photo'>) encountered during serialization".
2

If you use app-engine-patch it will automatically declare the _meta attribute for you, and then you can use django.core.serializers as you would normally do on django models (as in sledge's code).

App-engine-patch has some other cool features such has an hybrid authentication (django + google accounts), and the admin part of django works.

2 Comments

what's the difference between app-engine-patch vs google-app-engine-django vs the django version shipped with app engine python sdk? From what I understand, app-engine-patch is more complete?
I haven't tried the version of django on app engine, but I think it's integrated as is. google-app-engine-django if I'm not mistaken tries to make django's model work with app-engine (with some limitations). app-engine-patch uses directly app-engine models, they just add some minore stuffs to it. There is a comparison between the two on their website.
2

Mtgred's answer above worked wonderfully for me -- I slightly modified it so I could also get the key for the entry. Not as few lines of code, but it gives me the unique key:

class DictModel(db.Model):
def to_dict(self):
    tempdict1 = dict([(p, unicode(getattr(self, p))) for p in self.properties()])
    tempdict2 = {'key':unicode(self.key())}
    tempdict1.update(tempdict2)
    return tempdict1

Comments

2

I've extended the JSON Encoder class written by dpatru to support:

  • Query results properties (e.g. car.owner_set)
  • ReferenceProperty - recursively turn it into JSON
  • Filtering properties - only properties with a verbose_name will be encoded into JSON

    class DBModelJSONEncoder(json.JSONEncoder):
        """Encodes a db.Model into JSON"""
    
        def default(self, obj):
            if (isinstance(obj, db.Query)):
                # It's a reference query (holding several model instances)
                return [self.default(item) for item in obj]
    
            elif (isinstance(obj, db.Model)):
                # Only properties with a verbose name will be displayed in the JSON output
                properties = obj.properties()
                filtered_properties = filter(lambda p: properties[p].verbose_name != None, properties)
    
                # Turn each property of the DB model into a JSON-serializeable entity
                json_dict = dict([(
                        p,
                        getattr(obj, p)
                            if (not isinstance(getattr(obj, p), db.Model))
                            else
                        self.default(getattr(obj, p)) # A referenced model property
                    ) for p in filtered_properties])
    
                json_dict['id'] = obj.key().id() # Add the model instance's ID (optional - delete this if you do not use it)
    
                return json_dict
    
            else:
                # Use original JSON encoding
                return json.JSONEncoder.default(self, obj)
    

Comments

2

As mentioned by https://stackoverflow.com/users/806432/fredva, the to_dict works great. Here is my code i'm using.

foos = query.fetch(10)
prepJson = []

for f in foos:
  prepJson.append(db.to_dict(f))

myJson = json.dumps(prepJson))

1 Comment

yes, and there is also a "to_dict" on Model...this function is the key to making this whole problem as trivial as it should be. It even works for NDB with "structured" and "repeated" properties!
1

There's a method, "Model.properties()", defined for all Model classes. It returns the dict you seek.

from django.utils import simplejson
class Photo(db.Model):
  # ...

my_photo = Photo(...)
simplejson.dumps(my_photo.properties())

See Model properties in the docs.

1 Comment

Some objects aren't "JSON serializable": TypeError: <google.appengine.ext.db.StringProperty object at 0x4694550> is not JSON serializable
1

These APIs (google.appengine.ext.db) are no longer recommended. Apps that use these APIs can only run in the App Engine Python 2 runtime and will need to migrate to other APIs and services before migrating to the App Engine Python 3 runtime. To know more: click here

Comments

0

To serialize a Datastore Model instance you can't use json.dumps (haven't tested but Lorenzo pointed it out). Maybe in the future the following will work.

http://docs.python.org/2/library/json.html

import json
string = json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
object = json.loads(self.request.body)

4 Comments

the question is about converting an AppEngine Datastore Model instance to JSON. You solution is only about converting a Python dictionary to JSON
@tunedconsulting I haven't tried serializing a Datastore Model instance with json.dumps but assume it would work with any object. A bug report should be submitted if it isn't as the documentation states that json.dumps takes an object as parameter. It is added as a comment with only re comment that it didn't exist in 2009. Added this answer because it seems a bit out dated but if it would not work then I'm happy to remove it.
If you try to json.dumps an entity object or a model class you get TypeError: 'is not JSON serializable' <Object at 0x0xxxxxx>. GAE's Datastore has its own datatypes (for dates for example). The current right answer, tested and working, is the one from dmw that transforms some problematic datatypes into serializable ones.
@tunedconsulting Thank you for your input on this, I will update my answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.