Django rest framework db query slow

Question

I have a Django model with 16 DecimalFields.
I have created a ListAPIView to fetch this data.
With ~5000 instances of this model in my db, the GET request to this api view is taking more than 15 seconds.

Model:

class MyModel(models.Model):
    f1 = models.DecimalField (max_digits=8, decimal_places=3)
    f2 = models.DecimalField (max_digits=8, decimal_places=3)
    f3 = models.DecimalField (max_digits=8, decimal_places=3)
    f4 = models.DecimalField (max_digits=8, decimal_places=3)
    f5 = models.DecimalField (max_digits=8, decimal_places=3)
    f6 = models.DecimalField (max_digits=8, decimal_places=3)
    f7 = models.DecimalField (max_digits=8, decimal_places=3)
    f8 = models.DecimalField (max_digits=8, decimal_places=3)
    f9 = models.DecimalField (max_digits=8, decimal_places=3)
    f10 = models.DecimalField (max_digits=8, decimal_places=3)
    f11 = models.DecimalField (max_digits=8, decimal_places=3)
    f12 = models.DecimalField (max_digits=8, decimal_places=3)
    f13 = models.DecimalField (max_digits=8, decimal_places=3)
    f14 = models.DecimalField (max_digits=8, decimal_places=3)
    f15 = models.DecimalField (max_digits=8, decimal_places=3)
    f16 = models.DecimalField (max_digits=8, decimal_places=3)

Serializer:

class MyModelSerializer(serializers.ModelSerializer):
    class Meta:
        model = MyModel

I am not doing any filtering of the data. When I run the query in the django shell, it runs in a few ms. I assume serialization is the problem here. However, I am using the default ModelSerializer.

All of these fields are in the same table, so I don't see how there could be an N+1 issue here.

What can I do to further profile this issue? Should I expect my queries to be this slow with this many model instances?

Do you want to take all 5000 values to the client side? If not paginate and serialize only the required items. — Arun Ghosh
– Arun Ghosh, Commented Jul 21, 2015 at 9:37
There is nothing we can do but make blind guesses right now. Give us some eyes, give us the model and serializer. — Kevin Brown-Silva
– Kevin Brown-Silva, Commented Jul 21, 2015 at 11:50
@ArunGhosh I do want to take all 5000 values to the client side. As this data is only around 1mb, it seems like there should be a way to get from the db to client side in less than 15 seconds. — roob
– roob, Commented Jul 21, 2015 at 18:59
@KevinBrown Really? I thought my description of my model and serializer were straight forward enough, but I have updated the question. — roob
– roob, Commented Jul 21, 2015 at 19:02
@rawbeans That's... concerning. Usually non-query issues are because of customized models and serializers that do extra processing with the data. But if your serializers and models are bare (like your examples show), you're going to need to pull out an actual Python profiler. — Kevin Brown-Silva
– Kevin Brown-Silva, Commented Jul 21, 2015 at 19:05

Arpit Goyal · Accepted Answer · 2015-07-21 07:57:09Z

Rather than using ListAPIView try using a ModelViewSet instead.

Create a custom viewset for yourself which inherits only ListModelMixin

class ListModelViewSet(mixins.ListModelMixin, viewsets.GenericViewSet):
    pass

Now extend this to your viewset and write your queryset there

def get_queryset(self):
    Model.objects.all()

This is not a logic which can speed up things because ultimately all of them inherits the same thing but its worth a try.

If this also doesn't work... another possible reason can be that 5000 objects is a pretty large number. Shell does not return all the objects at once. It performs a query for Model.objects.all() but limits the number of objects to 20 followed by this ....(remaining elements truncated)...., so I guess this lessens the time as compared to the web view when it has to bring all the elements in the list.

Geo Jacob · Accepted Answer · 2015-07-21 07:37:30Z

0

This is because of serializer, you can improve the performance by using Model.objects.values() instaed of using serializer, This will return the same data very quickly.

answered Jul 21, 2015 at 7:37

Geo Jacob

6,0291 gold badge39 silver badges46 bronze badges

1 Comment

roob Over a year ago

I changed the queryset from Model.objects.all() to Model.objects.values() and the query is still just as slow.

Arun Ghosh · Accepted Answer · 2015-07-21 07:53:45Z

0

Install django-toolbar and see Query time and CPU time. If query is taking time and is doing some lazy loading check prefetch_related & select_related.

answered Jul 21, 2015 at 7:53

Arun Ghosh

7,7741 gold badge28 silver badges38 bronze badges

1 Comment

roob Over a year ago

Query time is 90ms, CPU time is 26000ms. Also, im not sure how prefetch_related and select_related are useful here because all fields are within the same table

Collectives™ on Stack Overflow

Django rest framework db query slow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related