I need to export multiple plots with Python and Matplotlib but I have issues with the memory management.
I'm using Jupyter Notebook in Windows and the code is structured as shown in the working example here below. Basically, I'm generating a daily plot over a period of several years.
Following the memory usage I can see that usually for the first 4 "months" the memory usage is fairly stable but then it increases more than 15MB for each "day". With the complete code this leads to a memory error.
I tried to use plt.close() as suggested here. I also tried plt.close(fig), plt.close('all') and to manually delete some variables with del. Finally I've put gc.collect in the loop. Nothing seems to have any effect.
import numpy as np
import matplotlib.pyplot as plt
import gc
sz = 1000
i=0
for year in [2014,2015,2016,2017]:
for month in [1,2,3,4,5,9,10,11,12]:
for day in range(1,30):
# Plot
fig, ((ax1,ax2),(ax3,ax4)) = plt.subplots(2,2,figsize=(14,10))
x1 = np.arange(1,sz+1,1)
y1 = np.random.random(sz)
ax1.scatter(x1,y1)
ax2.scatter(x1,y1)
ax3.scatter(x1,y1)
ax4.scatter(x1,y1)
filename = 'test/test_output{}{}{}.png'.format(year,month,day)
plt.tight_layout()
plt.savefig(filename, bbox_inches='tight')
#plt.close(fig)
#plt.close('all')
plt.close()
del x1, y1, ax1, ax2, ax3, ax4, fig
# counter
i=i+1
# Monthly operations
print(filename,i)
gc.collect()
# Triggered outside the loops
gc.collect()
On the other hand, if I interrupt the code (or I wait for the "Memory error" to occur) and then I run manually gc.collect, the memory is cleaned right away.
Why gc.collect does not have any effect in the nested loops?
Is there a way to improve the memory management for this code?
python scipt_name.py. How do you measure memory usage?plt.close()works as expected in this case. So the problem is with IPython and I still cannot find a solution. For checking the memory usage here I simply watch the Task Manager.plt.ioff()when in IPython?