Python memory of list in "for" loop

Question

Using some function I create a list of million items within and check memory consumption

print ('Memory at start: {0} Mb'.format(memory_usage_psutil()))
my_list = people_list(1000000)
print ('Memory at end: {0} Mb'.format(memory_usage_psutil()))

What I get is:

Memory at start: 100.90625 Mb
Memory at end: 403.8125 Mb

So creating a list and keeping it in memory takes ~300Mb.

For example I want to count amount of items in PART of my_list WITHOUT using len()

count = 0
additional_list = my_list[0:500000]
for item in additional_list:
    count += 1
    if count == 499999:
        print ('Memory in loop: {0} Mb'.format(memory_usage_psutil()))

and results confused me.

Memory at start: 100.90625 Mb
Memory in loop: 403.6953125 Mb
Memory at end: 403.6953125 Mb

I expected that creating additional list will increase memory consumption by 150Mb but it did not happen. Can you explain why? I mean id's are different

In [30]: id(my_list)
Out[30]: 2672641621704

In [31]: id(additional_list)
Out[31]: 2672642905864

and same results come from:

for item in my_list[0:500000]:
    count += 1
    if count == 499999:
        print ('Memory in loop: {0} Mb'.format(memory_usage_psutil()))

Meitham · Accepted Answer · 2016-07-10 19:39:28Z

2

It's true that your two lists are different objects and have different addresses in memory. However, the individual objects in the list are the same. Using a slice basically gave you a view on some elements in your list.

You can confirm that with::

>>> map(id, my_list)[0:500000] == map(id, additional_list)

answered Jul 10, 2016 at 19:39

Meitham

9,7905 gold badges37 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Alex Hall Over a year ago

The list still contains 500000 addresses to those objects which should cost at least a few MB.

Bakuriu Over a year ago

Your last example fails for me. In fact map(id, some_list) == map(id, some_list) is always False given that you are creating two different map objects.

Meitham Over a year ago

@Bakuriu map does not give you a map object, it is a python list of long, which the == operator will compare by contents. You might have map overshadowed by something else in your shell?

Bakuriu Over a year ago

>>> map(id, [1,2,3]) <map object at 0x7f6f6b384a58> You are probably using an outdated version of python.

Meitham Over a year ago

@Bakuriu indeed, I'm using outdated python2, I didn't realise map is lazy in python3, but that explains why == is False in your version and True in mine :-D

Collectives™ on Stack Overflow

Python memory of list in "for" loop

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related