5

I need to export multiple plots with Python and Matplotlib but I have issues with the memory management.

I'm using Jupyter Notebook in Windows and the code is structured as shown in the working example here below. Basically, I'm generating a daily plot over a period of several years.

Following the memory usage I can see that usually for the first 4 "months" the memory usage is fairly stable but then it increases more than 15MB for each "day". With the complete code this leads to a memory error.

I tried to use plt.close() as suggested here. I also tried plt.close(fig), plt.close('all') and to manually delete some variables with del. Finally I've put gc.collect in the loop. Nothing seems to have any effect.

import numpy as np
import matplotlib.pyplot as plt
import gc

sz = 1000
i=0
for year in [2014,2015,2016,2017]:
    for month in [1,2,3,4,5,9,10,11,12]:
        for day in range(1,30):

            # Plot
            fig, ((ax1,ax2),(ax3,ax4)) = plt.subplots(2,2,figsize=(14,10))

            x1 = np.arange(1,sz+1,1)
            y1 = np.random.random(sz)
            ax1.scatter(x1,y1)
            ax2.scatter(x1,y1)
            ax3.scatter(x1,y1)
            ax4.scatter(x1,y1)

            filename = 'test/test_output{}{}{}.png'.format(year,month,day)
            plt.tight_layout()
            plt.savefig(filename, bbox_inches='tight')

            #plt.close(fig)            
            #plt.close('all')
            plt.close()

            del x1, y1, ax1, ax2, ax3, ax4, fig

            # counter
            i=i+1

        # Monthly operations
        print(filename,i)
        gc.collect()

# Triggered outside the loops
gc.collect()

On the other hand, if I interrupt the code (or I wait for the "Memory error" to occur) and then I run manually gc.collect, the memory is cleaned right away.

Why gc.collect does not have any effect in the nested loops?

Is there a way to improve the memory management for this code?

5
  • What happens if you run as a script instead of in a notebook? I can imagine her notebook doing funky things when given hundreds of plots even if you close them. Commented Jul 23, 2018 at 14:32
  • I tried to run the script with Spyder and it happens the same (Python 3.6.4 - Spyder 3.2.6 - IPython 6.2.1). Commented Jul 23, 2018 at 14:58
  • 1
    "As script" means outside of IPython or similar, i.e. in the console run python scipt_name.py. How do you measure memory usage? Commented Jul 23, 2018 at 16:54
  • 1
    Ok, I tried to run the code as a script and the memory usage is kept correcly under control. It seems that plt.close() works as expected in this case. So the problem is with IPython and I still cannot find a solution. For checking the memory usage here I simply watch the Task Manager. Commented Jul 24, 2018 at 17:38
  • 1
    What happens if you turn interactive mode off, plt.ioff() when in IPython? Commented Jul 28, 2018 at 11:08

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.