matplotlib-devel Mailing List for matplotlib

Thanks, John!

Calling ax.autoscale_view after ax.relim makes the script work.

Best regards,
Pearu

Hi,

Ben Root gave me a bug report that pcolormesh (and hence QuadMesh)
were not respecting zorder. This turns out to be due to the fact that
kwargs are not being forwarded on as appropriate.  This is easy enough
to fix and make work, but I wanted to first ask for any insight on the
following "helpful" comment in axes.py:

       collection = mcoll.QuadMesh(
            Nx - 1, Ny - 1, coords, showedges,
            antialiased=antialiased, shading=shading)  # kwargs are not used

(It's be great if this comment gave some actual reasoning rather than
stating the obvious).

Anyone know if there's an explicit design choice for QuadMesh not
taking kwargs, or is it just an omission?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

Thanks, John!

Adding ax.autoscale_view after ax.relim makes the script work correctly.

Best regards,
Pearu

On Fri, May 28, 2010 at 1:37 PM, Pearu Peterson <pe...@ce...> wrote:

> While the new data is plotted correctly, the plot shows fixed axes
> from the first plot call. What I am doing wrong?

ax.relim() causes the data limits to be updated based on the current
objects it contains, ax.autoscale_view() causes the view limits to be
updated based on the data limits, and fig.canvas.draw() forces a
redraw.

JDH

On Fri, May 28, 2010 9:15 pm, John Hunter wrote:
> On Fri, May 28, 2010 at 1:04 PM, Pearu Peterson <pe...@ce...> wrote:
>
>> Regarding reusing existing line --- I have understood that this
>> will work only if the length of the line data does not change.
>
> This is not correct -- you can change the line length with calls to
> set_data
>
>> In my case the data grows as more data points are acquired and I have
>> not figured out how to make axes to set new limits after changing
>> the line data.
>
> ax.relim()

Ok, very good. However, it does not seem to have effect. Consider
the following example:

#
import numpy
from numpy.testing.utils import memusage
import matplotlib
matplotlib.use('GTKAgg')

import matplotlib.pyplot as plt

fig = plt.figure()
axes1 = fig.add_subplot( 111 )

def animate():
    x = [0]
    while 1:
        y = numpy.random.rand (len (x))
        if 1:
            # updating line in place
            if not axes1.lines:
                line, = axes1.plot(x, y, 'b')
            else:
                line.set_data(x, y)
                # relim does not have effect in updating axes
                axes1.relim()
        else:
            # demonstrates expected behaviour, has leakage w/o Mike patch
            for line in axes1.lines:
                line.remove()
            line, = axes1.plot(x, y, 'b')
        fig.canvas.draw()
        print memusage ()/(1024.0*1024.0),"MB", len (axes1.lines), len(x)
        x.append(x[-1]+1)

import gobject
print 'adding idle'
gobject.idle_add(animate)
print 'showing'
plt.show()
#eof

While the new data is plotted correctly, the plot shows fixed axes
from the first plot call. What I am doing wrong?

Pearu

On 05/28/2010 08:04 AM, Pearu Peterson wrote:
> On Fri, May 28, 2010 5:12 pm, John Hunter wrote:
>
>> Hey Pearu -- thanks for the report.  We'll try and track down and fix
>> this leak.  In the interim, would an acceptable work around for you be
>> to *reuse* an existing line by calling set_data on it.  That way you
>> wouldn't have to do the add/remove that is causing your leak.  Have
>> you confirmed this leak on various backends (eg Agg, PDF, PS)?
>
> No, I haven't but I can try it.
>
> Regarding reusing existing line --- I have understood that this
> will work only if the length of the line data does not change.

Not so.

> In my case the data grows as more data points are acquired and I have
> not figured out how to make axes to set new limits after changing
> the line data.

lineobj = plot([0], [0])[0]
ax = gca()
x = np.arange(10)
y = 20 * np.sin(x)
lineobj.set_data(x, y)
xy = np.concatenate((x[:,np.newaxis], y[:,np.newaxis]), axis=1)
ax.update_datalim(xy)
ax.autoscale_view()
draw()


Eric

>
> Currently I am using a work around where the axes are cleared
> after every 60 seconds - this seems to keep memory usage under control.
>
> It seems that Mike has resolved the problem. I'll try the latest
> SVN..
>
> Thanks!
> Pearu
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

On Fri, May 28, 2010 at 1:04 PM, Pearu Peterson <pe...@ce...> wrote:

> Regarding reusing existing line --- I have understood that this
> will work only if the length of the line data does not change.

This is not correct -- you can change the line length with calls to set_data

> In my case the data grows as more data points are acquired and I have
> not figured out how to make axes to set new limits after changing
> the line data.

ax.relim()


Cheers,
JDH

On Fri, May 28, 2010 5:12 pm, John Hunter wrote:

> Hey Pearu -- thanks for the report.  We'll try and track down and fix
> this leak.  In the interim, would an acceptable work around for you be
> to *reuse* an existing line by calling set_data on it.  That way you
> wouldn't have to do the add/remove that is causing your leak.  Have
> you confirmed this leak on various backends (eg Agg, PDF, PS)?

No, I haven't but I can try it.

Regarding reusing existing line --- I have understood that this
will work only if the length of the line data does not change.
In my case the data grows as more data points are acquired and I have
not figured out how to make axes to set new limits after changing
the line data.

Currently I am using a work around where the axes are cleared
after every 60 seconds - this seems to keep memory usage under control.

It seems that Mike has resolved the problem. I'll try the latest
SVN..

Thanks!
Pearu

There is a fix in r8341.  It passes the regression tests, and all of the 
event handling examples I tried seem to still work.

It seems that many places in matplotlib were never disconnecting 
callbacks, and these callbacks keep references to the destination 
objects alive.

Unfortunately, it's not quite obvious where the "disconnect" calls 
should be added -- the lifetime of  objects isn't very symmetrical.  For 
example, the "units" callback is set up by Lines2D inside of its 
"set_axes" method, but there is no "remove_axes" method in which to put 
the disconnect.  Tracking down all of the ways in which a line could be 
removed from an axes seems daunting.

Instead, my solution is to store weak references to the methods stored 
in the CallbackRegistry -- that way the CallbackRegistry won't leak 
references like it does now.  Since the Python stdlib weakref module 
doesn't directly support weak references to bound methods, the whole 
thing is a bit hairy -- but I think it's a more permanent solution than 
trying to ensure that all callbacks get explicitly disconnected.

As this change is rather fundamental and may have unintended 
consequences, please play with it in your contexts and let me know if 
you see anything strange.

Mike

On 05/28/2010 10:47 AM, Michael Droettboom wrote:
> I'm on to something -- some callbacks are being created that are never
> disconnected.
>
> In Line2D.set_axes:
>
>     self._xcid = ax.xaxis.callbacks.connect('units', self.recache_always)
>
> gets called twice.  This is problematic because the id of the first
> connection is simply lost.  Also, there doesn't seem to be any code to
> attempt to remove either of them.
>
> I'm looking into it further -- forcibly deleting these callbacks reduces
> the reference count on the line object, but doesn't seem to completely
> eliminate the leak.
>
> Mike
>
> On 05/28/2010 10:12 AM, John Hunter wrote:
>    
>> On Fri, May 28, 2010 at 3:18 AM, Pearu Peterson<pe...@ce...>   wrote:
>>
>>      
>>> Hi,
>>>
>>> In an application that updates a plot with
>>> new experimental data, say, every second and the experiment
>>> can last hours, I have tried two approaches:
>>> 1) clear axes and plot new experimental data - this is
>>> slow and takes too much cpu resources.
>>> 2) remove lines and plot new experimental data - this is
>>> fast enough but unfortunately there seems to be a memory
>>> leakage, the application runs out of memory.
>>>
>>> Here follows a simple script that demonstrates the
>>> leakage problem:
>>>
>>> #
>>> import numpy
>>> from numpy.testing.utils import memusage
>>> import matplotlib.pyplot as plt
>>> x = range (1000)
>>> axes1 = plt.figure().add_subplot( 111 )
>>> y = numpy.random.rand (len (x))
>>> while 1:
>>>      if 1:
>>>          # leakage
>>>          for line in axes1.lines:
>>>              if line.get_label ()=='data':
>>>                  line.remove()
>>>      else:
>>>          # no leak, but slow
>>>          axes1.clear()
>>>      axes1.plot(x, y, 'b', label='data')
>>>      print memusage (), len (axes1.lines)
>>> #eof
>>>
>>> When running the script, the memory usage
>>> is increasing by 132 kbytes per iteration, that is,
>>> with an hour this example application will consume
>>> 464MB RAM while no new data has been generated. In real
>>> application this effect will be even worse.
>>>
>>> So, I am looking for an advice how to avoid
>>> this memory leakage without clearing axes.
>>>
>>>        
>> Hey Pearu -- thanks for the report.  We'll try and track down and fix
>> this leak.  In the interim, would an acceptable work around for you be
>> to *reuse* an existing line by calling set_data on it.  That way you
>> wouldn't have to do the add/remove that is causing your leak.  Have
>> you confirmed this leak on various backends (eg Agg, PDF, PS)?
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Matplotlib-devel mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>>
>>      
>
>    


-- 
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

On Fri, May 28, 2010 at 10:52 AM, Jason Grout
<jas...@cr...> wrote:
> I just noticed here:
>
> http://matplotlib.sourceforge.net/api/axes_api.html#matplotlib.axes.Axes.set_xlim
>
> that the keywords that are documented for set_xlim are actually the
> keywords for set_ylim.  Surely this is just a copy-paste error.

Surely. Fixed in SVN, thanks for the report.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

I just noticed here:

http://matplotlib.sourceforge.net/api/axes_api.html#matplotlib.axes.Axes.set_xlim

that the keywords that are documented for set_xlim are actually the 
keywords for set_ylim.  Surely this is just a copy-paste error.

Thanks,

Jason

I'm on to something -- some callbacks are being created that are never 
disconnected.

In Line2D.set_axes:

   self._xcid = ax.xaxis.callbacks.connect('units', self.recache_always)

gets called twice.  This is problematic because the id of the first 
connection is simply lost.  Also, there doesn't seem to be any code to 
attempt to remove either of them.

I'm looking into it further -- forcibly deleting these callbacks reduces 
the reference count on the line object, but doesn't seem to completely 
eliminate the leak.

Mike

On 05/28/2010 10:12 AM, John Hunter wrote:
> On Fri, May 28, 2010 at 3:18 AM, Pearu Peterson<pe...@ce...>  wrote:
>    
>> Hi,
>>
>> In an application that updates a plot with
>> new experimental data, say, every second and the experiment
>> can last hours, I have tried two approaches:
>> 1) clear axes and plot new experimental data - this is
>> slow and takes too much cpu resources.
>> 2) remove lines and plot new experimental data - this is
>> fast enough but unfortunately there seems to be a memory
>> leakage, the application runs out of memory.
>>
>> Here follows a simple script that demonstrates the
>> leakage problem:
>>
>> #
>> import numpy
>> from numpy.testing.utils import memusage
>> import matplotlib.pyplot as plt
>> x = range (1000)
>> axes1 = plt.figure().add_subplot( 111 )
>> y = numpy.random.rand (len (x))
>> while 1:
>>     if 1:
>>         # leakage
>>         for line in axes1.lines:
>>             if line.get_label ()=='data':
>>                 line.remove()
>>     else:
>>         # no leak, but slow
>>         axes1.clear()
>>     axes1.plot(x, y, 'b', label='data')
>>     print memusage (), len (axes1.lines)
>> #eof
>>
>> When running the script, the memory usage
>> is increasing by 132 kbytes per iteration, that is,
>> with an hour this example application will consume
>> 464MB RAM while no new data has been generated. In real
>> application this effect will be even worse.
>>
>> So, I am looking for an advice how to avoid
>> this memory leakage without clearing axes.
>>      
> Hey Pearu -- thanks for the report.  We'll try and track down and fix
> this leak.  In the interim, would an acceptable work around for you be
> to *reuse* an existing line by calling set_data on it.  That way you
> wouldn't have to do the add/remove that is causing your leak.  Have
> you confirmed this leak on various backends (eg Agg, PDF, PS)?
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>    


-- 
Michael Droettboom
Science Software Branch
Space Telescope Science Institute
Baltimore, Maryland, USA

On Fri, May 28, 2010 at 3:18 AM, Pearu Peterson <pe...@ce...> wrote:
>
> Hi,
>
> In an application that updates a plot with
> new experimental data, say, every second and the experiment
> can last hours, I have tried two approaches:
> 1) clear axes and plot new experimental data - this is
> slow and takes too much cpu resources.
> 2) remove lines and plot new experimental data - this is
> fast enough but unfortunately there seems to be a memory
> leakage, the application runs out of memory.
>
> Here follows a simple script that demonstrates the
> leakage problem:
>
> #
> import numpy
> from numpy.testing.utils import memusage
> import matplotlib.pyplot as plt
> x = range (1000)
> axes1 = plt.figure().add_subplot( 111 )
> y = numpy.random.rand (len (x))
> while 1:
>    if 1:
>        # leakage
>        for line in axes1.lines:
>            if line.get_label ()=='data':
>                line.remove()
>    else:
>        # no leak, but slow
>        axes1.clear()
>    axes1.plot(x, y, 'b', label='data')
>    print memusage (), len (axes1.lines)
> #eof
>
> When running the script, the memory usage
> is increasing by 132 kbytes per iteration, that is,
> with an hour this example application will consume
> 464MB RAM while no new data has been generated. In real
> application this effect will be even worse.
>
> So, I am looking for an advice how to avoid
> this memory leakage without clearing axes.

Hey Pearu -- thanks for the report.  We'll try and track down and fix
this leak.  In the interim, would an acceptable work around for you be
to *reuse* an existing line by calling set_data on it.  That way you
wouldn't have to do the add/remove that is causing your leak.  Have
you confirmed this leak on various backends (eg Agg, PDF, PS)?

On 05/28/2010 09:18 AM, Pearu Peterson wrote:
> , say, every second and the experiment
> can last hours, I have tried two approaches:
> 1) clear axes and plot new experimental data - this is
> slow and takes too much cpu resources.
> 2) remove lines and plot new experimental data - this is
> fast enough but unfortunately there seems to be a memory
> leakage, the application runs out of memory.
>
Why don't you just update the exiting line with the new data, as shown 
in the animation examples in 
http://matplotlib.sourceforge.net/examples/animation/index.html ?

For example:

#
import numpy
from numpy.testing.utils import memusage
import matplotlib.pyplot as plt
x = range (1000)
fig = plt.figure()
axes1 = fig.add_subplot( 111 )
y = numpy.random.rand (len (x))
line = None
while 1:
     if not line:
         line, = axes1.plot(x, y, 'b', label='data')
     else:
         line.set_data(x,y)
	fig.canvas.draw()
     print memusage ()/(1024.0*1024.0),"MB", len (axes1.lines)
#eof


Regards,
João Silva

Hi,

In an application that updates a plot with
new experimental data, say, every second and the experiment
can last hours, I have tried two approaches:
1) clear axes and plot new experimental data - this is
slow and takes too much cpu resources.
2) remove lines and plot new experimental data - this is
fast enough but unfortunately there seems to be a memory
leakage, the application runs out of memory.

Here follows a simple script that demonstrates the
leakage problem:

#
import numpy
from numpy.testing.utils import memusage
import matplotlib.pyplot as plt
x = range (1000)
axes1 = plt.figure().add_subplot( 111 )
y = numpy.random.rand (len (x))
while 1:
    if 1:
        # leakage
        for line in axes1.lines:
            if line.get_label ()=='data':
                line.remove()
    else:
        # no leak, but slow
        axes1.clear()
    axes1.plot(x, y, 'b', label='data')
    print memusage (), len (axes1.lines)
#eof

When running the script, the memory usage
is increasing by 132 kbytes per iteration, that is,
with an hour this example application will consume
464MB RAM while no new data has been generated. In real
application this effect will be even worse.

So, I am looking for an advice how to avoid
this memory leakage without clearing axes.

I am using matplotlib from SVN.

Thanks,
Pearu