why is Django MultipleObjectsReturned Error hogging memory?

Question

I am using Django 1.9.10:

I have a model called Details has a unique_id column which is indexed:

 try:
     detail = Detail.objects.get(unique_id=UUID)
 except MultipleObjectsReturned as m:
     logger.error("uuid {} returned multiple objects - {}".format(UUID, str(m)))

Due to some error in code UUID=None this resulted in MultipleObjectsReturned error getting raised. but we noticed that almost 2-GB of memory is getting used up and it is slowing the system down a lot.

On printing str(m) in the error logs we found following error

MultipleObjectsReturned: get() returned more than one Details -- it returned 451424!

My Question is why is Django fetching so much data in memory just to raise an error? Django can just fetch the count?

I know I can use filter() to over come this issue but I am just surprised by this and want to understand why django is doing this?

sobolevn · Accepted Answer · 2017-03-07 10:02:25Z

3

Because that's how it's done internally:

def get(self, *args, **kwargs):
    """
    Perform the query and return a single object matching the given
    keyword arguments.
    """
    clone = self.filter(*args, **kwargs) # calling `filter()` inside
    if self.query.can_filter() and not self.query.distinct_fields:
        clone = clone.order_by()
    num = len(clone)
    if num == 1:
        return clone._result_cache[0]
    if not num:
        raise self.model.DoesNotExist(
            "%s matching query does not exist." %
            self.model._meta.object_name
        )
    raise self.model.MultipleObjectsReturned(
        "get() returned more than one %s -- it returned %s!" %
        (self.model._meta.object_name, num)
    )

See the full source here.

I guess you wonder why it can't just fetch the number? Because it will make two requests to the database instead of one. One request to fetch the count and one request to fetch the data.

How can you fix it? You may change your application logic to avoid this situation. Assert that UUID is not None. Or fetch count before making an actual query.

Docs about get: https://docs.djangoproject.com/en/dev/topics/db/queries/#retrieving-a-single-object-with-get

edited Mar 7, 2017 at 10:02

answered Mar 7, 2017 at 9:57

sobolevn

18.2k6 gold badges66 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Crazyshezy Over a year ago

Would it not be good idea to modified the get() with a limit() on the filter query. then just say that get() returned more than one result, skip the count part and avoid breaking the system by not bringing so many records into memory

bruno desthuilliers Over a year ago

@Crazyshezy you can submit a ticket and a patch to the Django project if you think it's a valuable improvement.

saurabh Over a year ago

@Crazyshezy so this is interesting. in 1.7 they had the limit as you are recommending. Not sure why they removed it later. check the code at github.com/django/django/blob/stable/1.7.x/django/db/models/…

RemcoGerlich · Accepted Answer · 2017-03-07 10:04:26Z

1

Because you use .get when you know you shouldn't get multiple objects.

Choosing to use it in a case where 451424 items are returned instead isn't something it was designed for.

answered Mar 7, 2017 at 10:04

RemcoGerlich

31.4k6 gold badges66 silver badges83 bronze badges

Comments

Marin · Accepted Answer · 2017-03-07 09:50:16Z

0

Because you have more then one with same ID in database. Rewrite your query to this: Detail.objects.filter(unique_id=UUID)

answered Mar 7, 2017 at 9:50

Marin

1,12114 silver badges34 bronze badges

Collectives™ on Stack Overflow

why is Django MultipleObjectsReturned Error hogging memory?

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related