Memory leak in python for loop even if I delete all variables at the end of each iteration

Question

The following two loops will use memory until I run out, but I can't figure out why. I am deleting all of the created variables at the end of each iteration and it still leaks.

!pip3 install cupy-cuda101
import cupy as cp
import numpy as np
from sklearn.preprocessing import PolynomialFeatures

xtrain = cp.asnumpy(cp.random.uniform(-1,1,size = (150000,50)))

for i in range(0,1000):
   weights = cp.random.uniform(-1,1,size = (1275,1000))
   
   for chunk in range(0,xtrain.shape[0],5000):
      xchunk = xtrain[chunk:chunk+5000,:]
      poly=PolynomialFeatures(interaction_only = True, include_bias = False)
      xchunk = cp.array(poly.fit_transform(xchunk))
      ranks = cp.matmul(xchunk,weights)
      del ranks, xchunk, poly
   del weights

xtrain is just float data as well, between -1 and 1.

All your del statements are pointless, since those variables are re-assigned on each iteration anyway. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jul 23, 2020 at 19:25
exactly, so how can it use more and more memory after each iteration? — xxanissrxx
– xxanissrxx, Commented Jul 23, 2020 at 19:26
No idea. You have to provide a minimal reproducible example to really expect anyone to be able to diagnose something like taht — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jul 23, 2020 at 19:31
Could you elaborate more on why you think there is a "leak"? Also note that CuPy uses the memory pool by default: docs.cupy.dev/en/latest/reference/memory.html — kmaehashi
– kmaehashi, Commented Jul 24, 2020 at 7:40

xxanissrxx · Accepted Answer · 2020-07-24 13:07:02Z

2

these lines insert at the end of each iteration fixed it:

    cp.get_default_memory_pool().free_all_blocks()
    cp.get_default_pinned_memory_pool().free_all_blocks()

answered Jul 24, 2020 at 13:07

xxanissrxx

1292 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Memory leak in python for loop even if I delete all variables at the end of each iteration

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related