Django - Return JSON, Error

Question

I got a View.py function that looks like this:

def GetAllCities(request):

cities = list(City.objects.all())
return HttpResponse(json.dumps(cities))

My City model looks like this

class City(models.Model):
    city = models.CharField()
    loc = models.CharField()
    population = models.IntegerField()
    state = models.CharField()
    _id = models.CharField()

    class MongoMeta:
        db_table = "cities"

    def __unicode__(self):
        return self.city

I am using a MongoDB that looks like this

{
   "_id" : ObjectId("5179837cbd7fe491c1f23227"),
   "city" : "ACMAR",
   "loc" : "[-86.51557, 33.584132]",
   "state" : "AL",
   "population" : 6055
}

I get the following error when trying to return the JSON from my GetAllCities function:

City ACMAR is not JSON serializable

So I tried this Instead:

def GetAllCities(request):

    cities = serializers.serialize("json", City.objects.all())
    return HttpResponse(cities)

And this works but It's very slow, it takes about 9 seconds(My database contains 30000 rows) Should it take this long or am I doing something wrong? I've built the same app in PHP, Rails and NodeJS. In PHP it takes on average 2000ms, NodeJS = 800ms, Rails = 5882ms and Django 9395ms. Im trying to benchmark here so I wonder if there is a way to optimize my Django code or is this as fast as it gets?

Mariusz Jamro · Accepted Answer · 2013-04-26 11:18:35Z

1

For sure you do not need to return ALL cities, as you probably won't display all 30000 rows anyway (at least in user-friendly way). Consider a solution where you return only cities within some range from requested location. Mongo supports geospatial indexes, so there should be no problem in doing that. There are also many tutorials over the internet how to perform spatial filtering in Django/MongoDB.
```
def GetAllCities(request, lon, lat):

    #Pseudo-code
    cities = City.objects.filterWithingXkmFromLonLat(lon, lat).all() 

    cities = serializers.serialize("json", cities) 
    return HttpResponse(cities)
```
If you really, really need all cities, consider caching the response. Location, name and population of cities are not things which change dynamically, in a matter of let's say seconds. Cache the result and recalculate only every hour, day or more. Django supports cache out of the box
```
@cache_page(60 * 60)
def GetAllCities(request):
   (...)
```

answered Apr 26, 2013 at 11:18

Mariusz Jamro

31.9k25 gold badges129 silver badges170 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JOSEFtw Over a year ago

This is a really good answer, but I'm benchmarking, see my answer :)

Mariusz Jamro Over a year ago

Even if you cut the processing time you actually haven't solved the problem. If you'll have two times more cities the time will be two times bigger.

Darwin · Accepted Answer · 2013-04-26 12:10:30Z

0

Another thing you can try to get a little more of speed is to get from db just the values you need and get the QuerySet to build the dictionary.

A simple query like this would work:

City.objects.all().values('id', 'city', 'loc', 'population', 'state')

Or you can put it in a manager:

class CitiesManager(models.Manager):

    class as_dict(self):
        return self.all().values('id', 'city', 'loc', 'population', 'state')

class City(models.Model):
    .... your fields here...

    objects = CitiesManager()

And then use it in your view as:

City.objects.as_dict()

answered Apr 26, 2013 at 12:10

Darwin

1,85915 silver badges17 bronze badges

1 Comment

JOSEFtw Over a year ago

Aaah very nice, vill try that! Thank you

Community · Accepted Answer · 2017-05-23 12:04:58Z

FOUND A SOLUTION

I am benchmarking with different methods, one method is to see how fast one language/framework is to select ALL rows in a database and return it as JSON. I found a solution now that speeds it up by half the time!

My new views.py

def GetAllCities(request):

    dictionaries = [ obj.as_dict() for obj in City.objects.all() ]
    return HttpResponse(json.dumps({"Cities": dictionaries}), content_type='application/json')

And my new model

class City(models.Model):
    city = models.CharField()
    loc = models.CharField()
    population = models.IntegerField()
    state = models.CharField()
    _id = models.CharField()
def as_dict(self):
        return {
            "id": self.id,
            "city": self.city,
            "loc": self.loc,
            "population": self.population,
            "state": self.state
            # other stuff
        }

    class MongoMeta:
         db_table = "cities"

def __unicode__(self):
        return self.city

Found the solution here

Collectives™ on Stack Overflow

Django - Return JSON, Error

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related