Why python list slice assignment eats memory?

Question

I'm fighting a memory leak in a Python project and spent much time on it already. I have deduced the problem to a small example. Now seems like I know the solution, but I can't understand why.

import random

def main():
    d = {}
    used_keys = []
    n = 0
    while True:
        # choose a key unique enough among used previously
        key = random.randint(0, 2 ** 60)
        d[key] = 1234 # the value doesn't matter
        used_keys.append(key)
        n += 1
        if n % 1000 == 0:
            # clean up every 1000 iterations
            print 'thousand'
            for key in used_keys:
                del d[key]
                used_keys[:] = []
                #used_keys = []

if __name__ == '__main__':
    main()

The idea is that I store some values in the dict d and memorize used keys in a list to be able to clean the dict from time to time.

This variation of the program confidently eats memory never returning it back. If I use alternative method to „clear” used_keys that is commented in the example, all is fine: memory consumption stays at constant level.

Why?

Tested on CPython and many linuxes.

How do you know for sure it never returns it? It might just be that the OS never asks for it back. — detly
– detly, Commented Jul 30, 2010 at 8:25
Shouldn't clearing used_keys be outside of the for key in used_keys loop? — adamk
– adamk, Commented Jul 30, 2010 at 8:27
>The idea is that I store some values in the dict d and memorize used keys in a list to be able to clean the dict from time to time. Why not use just d.keys()? It will be same list of keys. — Daniel Kluev
– Daniel Kluev, Commented Jul 30, 2010 at 8:28
@adamk See a comment to the accepted reply. @Daniel and @gnibbler Its just a model, if it were stand-alone code, I wouldn't use such odd methods. — nkrkv
– nkrkv, Commented Jul 30, 2010 at 8:43

adamk · Accepted Answer · 2010-07-30 08:31:08Z

5

Here's the reason - the current method does not delete the keys from the dict (only one, actually). This is because you clear the used_keys list during the loop, and the loop exits prematurely.

The 2nd (commented) method, however, does work as you assign a new value to used_keys so the loop finishes successfully.

See the difference between:

>>> a=[1,2,3]
>>> for x in a:
...    print x
...    a=[]
...
1
2
3

and

>>> a=[1,2,3]
>>> for x in a:
...    print x
...    a[:] = []
...
1
>>>

answered Jul 30, 2010 at 8:31

adamk

47.1k7 gold badges52 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

nkrkv Over a year ago

Ah!! I'm stupid, stupid, stupid. I was so happy to reconstruct the memory leak in a small snippet… It is a sad mistake, of course. It doesn't represent my problem, I gonna continue hunting. But you're right with the answer on the original question. Thanks!

John La Rooy · Accepted Answer · 2010-07-30 08:38:04Z

0

Why wouldn't something like this work?

from itertools import count
import uuid

def main():
    d = {}
    for n in count(1):
        # choose a key unique enough among used previously
        key = uuid.uuid1()
        d[key] = 1234 # the value doesn't matter
        if n % 1000 == 0:
            # clean up every 1000 iterations
            print 'thousand'
            d.clear()

if __name__ == '__main__':
    main()

answered Jul 30, 2010 at 8:38

John La Rooy

306k54 gold badges378 silver badges514 bronze badges

Collectives™ on Stack Overflow

Why python list slice assignment eats memory?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related