Python list of Objects taking up too much memory

Question

I have the following code, that creates a million objects of a class foo:

for i in range(1000000):
    bar = foo()
    list_bar.append(bar)

The bar object is only 96 bytes, as determined by getsizeof(). However, the append step takes almost 8GB of ram. Once the code exits the loop, the ram usage drops to expected amounts (size of the list + some overhead ~103MB). Only while the loop is running does the ram usage skyrocket. Why does this happen? Any workarounds? PS: Using a generator is not an option, it has to be a list.

EDIT: xrange doesn't help, using Python 3. The memory usage stays high only during the loop execution, and drops after the loop is through. Could append have some non-obvious overhead?

Why is using xrange (a generator) instead of range not an option? A list with a million elements will take up a lot of RAM; there's no way around that. — sapi
– sapi, Commented Mar 11, 2015 at 11:48
list_bar = [foo() for _ in xrange(1000000)] will reduce the memory overhead, and make your code more idiomatic. — jonrsharpe
– jonrsharpe, Commented Mar 11, 2015 at 11:50
When you say "it has to be a list", do you mean that list_bar has to be a list? Or that the result of the for loop condition expression (i.e., range) must be a list? — Kevin
– Kevin, Commented Mar 11, 2015 at 11:51
Unless you provide more information, the question might get closed for not being clear. — thefourtheye
– thefourtheye, Commented Mar 11, 2015 at 11:53

Antti Haapala · Accepted Answer · 2017-11-24 10:03:18Z

7

Most probably this is due to some unintended cyclical references made by the foo() constructor; as normally Python objects will release memory instantly when the reference count drops to zero; now these would be freed later when the garbage collector gets a chance to run.

You can try to force the GC run after say 10000 iterations to see if it keeps the memory usage constant.

import gc
n = 1000000
list_bar = [ None ] * n
for i in range(n):
    list_bar[i] = foo()
    if i % 10000 == 0:
        gc.collect()

If this relieves memory pressure then the memory usage is because of some reference cycles.

The resizing of a list has some overhead. If you know how many elements, then you can create the list beforehand, e.g.:

list_bar = [ foo() for _ in xrange(1000000) ]

should know the size of the array and not need to resize it; or create the list filled with None:

n = 1000000
list_bar = [ None ] * n
for i in range(n):
    list_bar[i] = foo()

append should be using realloc to grow the list, but old memory ought to be released as soon as possible; and all in all the overhead of all memory allocated should not sum to 8G for a list that is 100 MB at the end; it can be possible that the operating system is miscalculating the memory used.

edited Nov 24, 2017 at 10:03

answered Mar 11, 2015 at 12:08

Antti Haapala

135k23 gold badges298 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Srivathsan Jayaraman Over a year ago

Hey, the length of the list is known. I already tried creating a list of dummy elements and replacing them as such, it doesn't help. I'm starting to think this has less to do with append and more to do with the foo class; the class inherits a third party module that might be causing problems. It's definitely not a miscalculation, tested on two separate computers.

Srivathsan Jayaraman Over a year ago

Thanks, turns out there was a cyclic reference in there.

mhawke · Accepted Answer · 2015-03-11 12:51:06Z

How are you measuring the memory usage?

I suspect your usage of a 3rd party module might be the cause. Perhaps the 3rd party module is temporarily using a lot of memory when initialised.

Besides, sys.getsizeof() is not an accurate indication of the memory used by an object.

For example:

from sys import getsizeof

class A(object):
    pass

class B(object):
    def __init__(self):
        self.big = 'a' * 1024*1024*1024    # approx. 1 GiB

>>> getsizeof(A)
976
>>> a = A()
>>> getsizeof(a)
64
>>> 
>>> getsizeof(B)
976
>>> b = B()
>>> getsizeof(b)
64
>>> getsizeof(b.big)
1073741873

After instantiating b = B(), top reports approx 1GiB resident memory usage. Obviously this is not reflected by getsizeof(b) which returns only 64 bytes.

Collectives™ on Stack Overflow

Python list of Objects taking up too much memory

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related