matplotlib-users Mailing List for matplotlib

>>>>> "Matthew" == Matthew O'Connnor <mat...@ca...> writes:

    Matthew> Hi, I was wondering if anybody else was seeing memory
    Matthew> leaks in the latest version (0.60.2).

    Matthew> I'm using the Agg backend to produce images for web
    Matthew> pages.

    Matthew> The release notes for 0.60.2 indicate that memory leaks
    Matthew> have been fixed.  Are there any known leaks left?

Yes, when I say memory leaks were fixed, I mean I identified and fixed
some, but am not claiming that all are fixed.  There are known memory
leaks.  For example, there is a small leak in freetype2 that was fixed
in freetype CVS but may not be released yet.  There appears to be a
leak in _backend_agg's write_png method, but I haven't traced this
sufficiently to find if it is in my code or in libpng or in zlib.

The whole memory leak situation has continued to improve as I've
migrated almost all of the extension code to pycxx, and I have a
number of unit tests to test leaks in various components.  For
example, there are no detectable leaks in the transforms module.

Below is a prototypical script that I use to diagnose and report
memory leaks.  Please use something like it when reporting leaks so I
get an idea of the magnitude of the problem (ie bytes per figure).
Also please provide your platform, matplotlib version, backend and as
much information about the versions of the libraries you are using:
freetype, png and zlib.  It would also help if you posted code so I
could look for any obvious coding errors vis-a-vis matplotlib.

Running the code below, I'm getting about 400 bytes / per / figure
(matplotlib-0.60.2 / agg on linux with freetype 2.17, libpng1.2 and
libz 1.1.4).  I know leaks of this size can cause troubles on an
application server, and in animated figures, and will continue to try
and track them down.  As I noted above, some of this results from the
libraries on which matplotlib depends, and so may be system and
version dependent.  If your numbers are much worse, it may be that you
are doing something wrong, ie, not properly closing, reusing figure,
etc.

Let me know what you find,
JDH

import os, sys, time
import matplotlib
matplotlib.use('Agg')
from matplotlib.matlab import *


def report_memory(i):
    pid = os.getpid()
    a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
    print i, '  ', a2[1],
    return int(a2[1].split()[1])

N = 100
for i in range(N):
    ind = arange(100)
    xx = rand(len(ind), 1)
    figure(1)
    plot(ind, xx[:,0])
    savefig('tmp%d.png' % i, dpi = 75)
    close(1)

    val = report_memory(i)
    if i==1: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes\n' % ((end-start)/float(N))

>>>>> "Jeff" == Jeff Whitaker <js...@fa...> writes:

    Jeff> John: I found that if I just call proj with all the lats and
    Jeff> lons at once (instead of once for each segment) I can speed
    Jeff> it up tremendously. Here's what I tried, using the new
    Jeff> LineCollection snippets you sent me, and the updated
    Jeff> matplotlib snapshot:

Yep - very good idea.  Your mistake with the line collection is how
you define the segments.  From matplotlib.collections.LineCollection
documentation

    segments is a sequence of ( line0, line1, line2), where linen =
    (x0, y0), (x1, y1), ... (xm, ym).  Each line can be a different
    length

That is, it is a sequence of sequences of xy tuples.  When you write
zip(xs.tolist(),ys.tolist()), you have a sequence of xy tuples.  If
you plotted this, you would have one giant, connected line, which is
not what you want.  You want a series of disconnected lines.  Thus you
need to keep track of the indices where each separate segment starts,
and split out the segments, as in the code below.

For future reference, you may also want to use PolyCollections if you
want to generate a map and you have a bunch of segments defined by
sequences of xy tuples with associated face colors.  

There was no significant difference between using bilinear
interpolation with antialiased drawing vs nearest neighbor
interpolation w/o aa, so I turned both back on.

And don't forget to try the new toolbar....

I noticed that the lat/lon lines don't precisely agree with the
colormap, eg around the Aleutian Islands the light blue is not
perfectly registered with the black lines.  Should I be concerned that
this reflects a problem in matplotlib, or is this kind of error
standard in the data you've provided?  I think this is related to the
pixel border that appears around some images, and is magnified when
interpolation is used because the top and right borders are not
defined when interpolating.  I'll continue to look into this.

Would it be OK if I used this example on the web page?  I like it!

Enjoy,
JDH

import cPickle
from matplotlib.matlab import *
from matplotlib.collections import LineCollection 
from proj import Proj
import Numeric

# standard parallels at 50 deg N, center longitued 107 deg W.
params = {}
params['proj'] = 'lcc'
params['R'] = 63712000
params['lat_1'] = 50
params['lat_2'] = 50
params['lon_0'] = -107
proj = Proj(params)
llcornerx, llcornery = proj(-145.5,1.)
params['x_0'] = -llcornerx # add cartesian offset so lower left corner = (0,0)
params['y_0'] = -llcornery
# create a Proj instance for desired map.
proj = Proj(params)

# set the default params for imshow
rc('image', origin='lower', cmap='jet')
ax = subplot(111)
nx = 349; ny = 277
dx = 32463.41; dy = 32463.41
xmax = (nx-1)*dx; ymax = (ny-1)*dy # size of domain to plot

C = cPickle.load( file('topodata.pickle','rb') )
im = ax.imshow(C, interpolation='bilinear',
               extent=(0, xmax, 0, ymax))

# ind is the index for the start of a new group
lons = []; lats = []; ind = []
i = 0  # the current ind
for line in file('wcl.txt'):
    if line.startswith('# -b'):
        ind.append(i)
        continue
    
    # lon/lat
    lon, lat = [float(val) for val in line.split('\t')]
    lons.append(lon)
    lats.append(lat)
    i += 1

xs, ys = proj(Numeric.array(lons),Numeric.array(lats))
#a sequence of xy tuples
segments = [zip(xs[i0:i1], ys[i0:i1]) for i0, i1 in zip(ind[:-1], ind[1:])]


collection = LineCollection(
    segments,     
    colors       = ( (0,0,0,1), ), # black              
    linewidths   = (1.5,),                              
    antialiaseds = (1,),)  # turn off aa for speed     
ax.add_collection(collection)
corners = (min(xs), min(ys)), (max(xs), max(ys))
ax.update_datalim( corners )
axis([0, xmax, 0, ymax])
ax.set_xticks([]) # no ticks
ax.set_yticks([])
title('Lambert Conformal Conic Projection')                  
#savefig('test')
show()      

On Tue, 13 Jul 2004, John Hunter wrote:

>
> OK, made some headway here.
>
....
>
> For the line part of the map, I extended the
> matplotlib.collections.LineCollection class to handle a sequence of
> lines, where each line is defined by a list of tuples (x0,y0), (x1,
> y1), ...  Thus all of your lines are handled by a single object,
> rather than having 1800+ separate line objects created in plot.
> Again, no python loops required.
>
> In the current form, the code takes about 1.15s to run on my machine
> and is about 30x faster than the original code you posted which
> includes the data loading part.  Nonetheless, the matplotlib part is
> much faster too, as you'll see when you interact with the data.
>

John:  I found that if I just call proj with all the lats and lons at 
once (instead of once for each segment) I can speed it up tremendously. 
Here's what I tried, using the new LineCollection snippets you sent me, 
and the updated matplotlib snapshot:


from matplotlib.matlab import *
from matplotlib.collections import LineCollection 
from proj import Proj
import Numeric, time

# open file with coastline data (from world coastline database).
wcl = open('wcl.txt')

# set up map projection parameters (lambert conformal conic,
# standard parallels at 50 deg N, center longitued 107 deg W.
params = {}
params['proj'] = 'lcc'
params['R'] = 63712000
params['lat_1'] = 50
params['lat_2'] = 50
params['lon_0'] = -107
proj = Proj(params)
llcornerx, llcornery = proj(-145.5,1.)
params['x_0'] = -llcornerx # add cartesian offset so lower left corner = (0,0)
params['y_0'] = -llcornery
# create a Proj instance for desired map.
proj = Proj(params)

# set the default params for imshow 
rc('image', origin='lower', cmap='jet')

ax = subplot(111) 
nx = 349; ny = 277 
dx = 32463.41; dy = 32463.41 
xmax = (nx-1)*dx; ymax = (ny-1)*dy # size of domain to plot

t1 = time.clock()
segnum = 0
lons = []
lats = []
for line in wcl:
     splitline = line.split()
     if splitline[0] != '#':
         lon = float(splitline[0])
 	lat = float(splitline[1])
 	lons.append(lon)
 	lats.append(lat)
xs, ys = proj(Numeric.array(lons),Numeric.array(lats))
minx, maxx = min(xs), max(xs) 
miny, maxy = min(ys), max(ys) 
xypts = zip(xs.tolist(),ys.tolist())

wcl.close() # close coastline file
# all args are sequences, length 1 in case of linewidths and 
# antialiased 
collection = LineCollection(segments     = xypts,
                             colors       = ( (0,0,0,1), ), # black
                             linewidths   = (1.5,),
                             antialiaseds = (0,),)  # turn off aa for speed 
ax.add_collection(collection) 
ax.update_datalim( ((minx, miny), (maxx, maxy)))

axis([0, xmax, 0, ymax]) 
ax.set_xticks([]) # no ticks 
ax.set_yticks([]) 
title('Lambert Conformal Conic Projection') 
t2 = time.clock()
print t2-t1
show()

But when I run it, I get the following error:

Traceback (most recent call last):
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_gtkagg.py", 
line 75, in callback
     self.draw()
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_gtkagg.py", 
line 42, in draw
     agg.draw()
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_agg.py", line 
291, in draw
     self.figure.draw(self.renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/figure.py", line 236, 
in draw
     for a in self.axes: a.draw(renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/axes.py", line 668, in 
draw
     c.draw(renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/collections.py", line 
286, in draw
     self._transOffset)
TypeError: CXX: type error.


-Jeff

  -- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

Hi,

I was wondering if anybody else was seeing memory leaks in the latest
version (0.60.2).

I'm using the Agg backend to produce images for web pages.

The release notes for 0.60.2 indicate that memory leaks have been fixed.
Are there any known leaks left?

Cheers,

Matthew O'Connor