matplotlib-users Mailing List for matplotlib

Thanks,
Ben, that makes sense although I think the angles I have can be considered
the direction 'to' and I want direction 'from' so adding a negative makes
them come from the direction I need.

I should add I don't really know what flip does so will ignore that.
thanks for the responses.



On Fri, Oct 14, 2011 at 9:32 AM, Eric Firing <ef...@ha...> wrote:

> On 10/13/2011 12:22 PM, Benjamin Root wrote:
>
>>
>>
>> On Thursday, October 13, 2011, questions anon <que...@gm...
>> <mailto:questions.anon@gmail.**com <que...@gm...>>> wrote:
>>  > Another quick question,
>>  > I noticed that the wind direction appeared back-to-front so I have
>> added a negative  to my equation as I am in the southern hemisphere. It
>> now looks correct but I wanted to check if this is the right way to do
>> this? I saw something called 'flip' for barb but nothing for quiver.
>>
>> It probably nothing to do with southern hemisphere but rather that it is
>> typical for wind data to be recorded and displayed as direction "from",
>> instead of direction "to".  If you think "flip" should be available for
>> quiver(), file a feature request.
>>
>
> It looks to me like the barb "flip" is for something else entirely--it is a
> right-hand versus left-hand sort of flip.  This would be inappropriate for
> quiver, which makes symmetric arrows.  I would strongly oppose any proposal
> for a kwarg to reverse the direction of the arrows.
>
> Eric
>
>
>
>> Ben Root
>>
>
>

On 10/13/2011 12:22 PM, Benjamin Root wrote:
>
>
> On Thursday, October 13, 2011, questions anon <que...@gm...
> <mailto:que...@gm...>> wrote:
>  > Another quick question,
>  > I noticed that the wind direction appeared back-to-front so I have
> added a negative  to my equation as I am in the southern hemisphere. It
> now looks correct but I wanted to check if this is the right way to do
> this? I saw something called 'flip' for barb but nothing for quiver.
>
> It probably nothing to do with southern hemisphere but rather that it is
> typical for wind data to be recorded and displayed as direction "from",
> instead of direction "to".  If you think "flip" should be available for
> quiver(), file a feature request.

It looks to me like the barb "flip" is for something else entirely--it 
is a right-hand versus left-hand sort of flip.  This would be 
inappropriate for quiver, which makes symmetric arrows.  I would 
strongly oppose any proposal for a kwarg to reverse the direction of the 
arrows.

Eric


>
> Ben Root

On Thursday, October 13, 2011, questions anon <que...@gm...>
wrote:
> Another quick question,
> I noticed that the wind direction appeared back-to-front so I have added a
negative  to my equation as I am in the southern hemisphere. It now looks
correct but I wanted to check if this is the right way to do this? I saw
something called 'flip' for barb but nothing for quiver.

It probably nothing to do with southern hemisphere but rather that it is
typical for wind data to be recorded and displayed as direction "from",
instead of direction "to".  If you think "flip" should be available for
quiver(), file a feature request.

Ben Root

Another quick question,
I noticed that the wind direction appeared back-to-front so I have added a
negative  to my equation as I am in the southern hemisphere. It now looks
correct but I wanted to check if this is the right way to do this? I saw
something called 'flip' for barb but nothing for quiver.
Thanks, see code below

               map = Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,

llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')

                x,y=map(*N.meshgrid(LON,LAT))
                u=-1*N.sin(WDIR*N.pi/180)# added negative to account for
southern hemisphere
                v=-1*N.cos(WDIR*N.pi/180)
                map.quiver(x,y,u, v)



On Wed, Oct 5, 2011 at 3:20 PM, questions anon <que...@gm...>wrote:

> thank you! of course!
>
>
> On Wed, Oct 5, 2011 at 3:08 PM, Eric Firing <ef...@ha...> wrote:
>
>> On 10/04/2011 05:40 PM, questions anon wrote:
>> > Excellent, thank you. That works for both quiver and barb.
>> > In regards to the shape that has to do with the netcdf file, wind
>> > direction variable also has lat and lon. I ended up needing to slice so
>> > I could skip a few points otherwise there was too many arrows on the
>> map.
>> > Below is the code that worked in case someone else needs help.
>>
>> Are you sure it worked correctly?  I don't think so.  See below.
>>
>> > Thanks again
>> >
>> > from netCDF4 import Dataset
>> > import numpy as N
>> > import pylab as plt
>> > from numpy import ma as MA
>> > from mpl_toolkits.basemap import Basemap
>> > import os
>> >
>> > ncfile=Dataset('E:/temp_winddir/IDZ00026_VIC_ADFD_Wind_Dir_SFC.nc',
>> > 'r+', 'NETCDF4')
>> > WDIR=ncfile.variables['Wind_Dir_SFC'][20,0::5,0::5]
>>
>> Don't you need to convert from degrees to radians?  sin and cos (used
>> below) expect radians, not degrees.  You can use N.deg2rad(WDIR) to
>> convert.
>>
>> > LAT=ncfile.variables['latitude'][0::5]
>> > LON=ncfile.variables['longitude'][0::5]
>> > TIME=ncfile.variables['time'][20]
>> > ncfile.close()
>> >
>> > map = Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>> >                llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>> > x,y=map(*N.meshgrid(LON,LAT))
>> > u=1*N.sin(WDIR)
>> > v=1*N.cos(WDIR)
>> > map.quiver(x,y,u, v)
>>
>> Assuming you do convert to radians the above will work for your mercator
>> projection but it will not align the vectors correctly with respect to
>> the parallels and meridians for other projections, so you should use
>> map.rotate_vector as in the basemap quiver demo.
>>
>> Eric
>>
>>
>> >
>> > #  otherwise use the barb tool - map.barbs(x,y,u, v)
>> >
>> > plt.title('Wind Direction')
>> > plt.show()
>>
>>
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2dcopy1
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>
>

On 10/13/2011 09:50 AM, Durrieu Jean-Louis wrote:
> Hi Eric, Fernando,
>
> thanks for your answers!
>
> On Oct 13, 2011, at 8:47 PM, Eric Firing wrote:
>
>> On 10/12/2011 10:47 PM, Durrieu Jean-Louis wrote:
>>
>> There is no compiled qt4 code, so installation should not require
>> pyqt4--and I don't think it does.  What led to your conclusion that
>> you needed it to install from source?
>>
>
> It s just that when I type "python setup.py build/--help/install",
> depending on whether pyqt4 is installed or not, the script detects
> that Qt is present or not (respectively), and pyside does not change
> that behaviour. I must admit I did not try to run my script even when
> Qt was not detected, maybe it works... But I guess there is a
> consequence to the detection of Qt at the build step?
>
> Sorry for not providing the exact output of the build step, I m on
> another computer. On my mac, the section of interest is: OPTIONAL
> BACKEND DEPENDENCIES libpng: 1.5.2 Tkinter: Tkinter: version not
> identified, Tk: 8.5, Tcl: 8.5 Gtk+: no * Building for Gtk+ requires
> pygtk; you must be able * to "import gtk" in your build/install
> environment Mac OS X native: yes Qt: no Qt4: Qt: 4.7.3, PyQt4: 4.8.5
> Cairo: no
>
> Here, it detected my pyqt4. But when it s not there, even if pyside
> is installed, I get a "Qt4: no" instead... Do you mean that it's
> still going to work, even it was not detected there?

Yes, it should work fine; as far as I know, nothing depends on the 
outcome of that auto-detection for qt4. The pyside support is always 
included in backend_qt4.py and backend_qt4agg.py, which are pure python, 
and are present regardless of whether pyqt4 itself is detected.

Eric

>
>> This does not make sense to me.  Assuming you are using bash, put
>> this in your .bashrc file:
>>
>> export QT_API=pyside
>>
>> Anything you execute in a terminal window opened after that change
>> should find the new environment variable.
>>
>
> Oh, sorry for that! I must have been confused, I think I only typed
> "QT_API=pyside" at the prompt, so I guess without the "export"
> (doh!), that was useless... Since the doc (somewhere) talked about
> "ETS environment variables", I thought there was something else to
> it. If not, then I ll have to learn my bash again :-)
>
> Not a big issue, anyway!
>
> Cheers!
>
> Jean-Louis
> ------------------------------------------------------------------------------
>
>
All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and
> makes sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________ Matplotlib-users
> mailing list Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

Hi Eric, Fernando,

thanks for your answers!

On Oct 13, 2011, at 8:47 PM, Eric Firing wrote:

> On 10/12/2011 10:47 PM, Durrieu Jean-Louis wrote:
> 
> There is no compiled qt4 code, so installation should not require 
> pyqt4--and I don't think it does.  What led to your conclusion that you 
> needed it to install from source?
> 

It s just that when I type "python setup.py build/--help/install", depending on whether pyqt4 is installed or not, the script detects that Qt is present or not (respectively), and pyside does not change that behaviour. I must admit I did not try to run my script even when Qt was not detected, maybe it works... But I guess there is a consequence to the detection of Qt at the build step?

Sorry for not providing the exact output of the build step, I m on another computer. On my mac, the section of interest is:
OPTIONAL BACKEND DEPENDENCIES
                libpng: 1.5.2
               Tkinter: Tkinter: version not identified, Tk: 8.5, Tcl: 8.5
                  Gtk+: no
                        * Building for Gtk+ requires pygtk; you must be able
                        * to "import gtk" in your build/install environment
       Mac OS X native: yes
                    Qt: no
                   Qt4: Qt: 4.7.3, PyQt4: 4.8.5
                 Cairo: no

Here, it detected my pyqt4. But when it s not there, even if pyside is installed, I get a "Qt4: no" instead... Do you mean that it's still going to work, even it was not detected there?

> This does not make sense to me.  Assuming you are using bash, put this 
> in your .bashrc file:
> 
> export QT_API=pyside
> 
> Anything you execute in a terminal window opened after that change 
> should find the new environment variable.
> 

Oh, sorry for that! I must have been confused, I think I only typed "QT_API=pyside" at the prompt, so I guess without the "export" (doh!), that was useless... Since the doc (somewhere) talked about "ETS environment variables", I thought there was something else to it. If not, then I ll have to learn my bash again :-)

Not a big issue, anyway!

Cheers! 

Jean-Louis

On 10/12/2011 10:47 PM, Durrieu Jean-Louis wrote:
> Dear all, matplotlib users and devs,
>
> I have been trying the latest announced version of Matplotlib
> (1.1.0). The new feature that attracted me is the PySide support.
>
> However, I tried to install it under Ubuntu 11.04, but I needed pyqt4
> installed in order to install it from source. Is there a way to
> install it with only pyside installed?

There is no compiled qt4 code, so installation should not require 
pyqt4--and I don't think it does.  What led to your conclusion that you 
needed it to install from source?

In any case, I opened
https://github.com/matplotlib/matplotlib/issues/521 for this.

>
> Once installed, I could uninstall pyqt4, and run my python scripts,
> with replacing pyqt4 by pyside. However, I am not sure what the "ETS
> environment variable" is (yes, it s related to the enthought tool
> suite, but what if I do not want it - for now), and I had to set the
> QT_API variable on the command line call to python: setting it as a
> "normal" environment variable does not work. So, to be able to use
> "only" pyside, I need to do:
>
> QT_API=pyside python myscript.py
>
> Is that the right way to do so? Is there any way of setting the
> QT_API more "definitely", and not having to change the call to
> python?

This does not make sense to me.  Assuming you are using bash, put this 
in your .bashrc file:

export QT_API=pyside

Anything you execute in a terminal window opened after that change 
should find the new environment variable.

Eric

>
> So far, my program (I really have only one at the moment) seems to
> work well, using QtCore, QtGui and phonon, except for phonon, where
> it seems that "seeking" in a file does not really work under
> Ubuntu... if anyone has an idea (I know, not the right mailing list,
> but just in case).
>
> At last, probably a question that will solve all the above ones: when
> is the new Matplotlib version going to be included in EPD? That's, by
> the way, mainly the reason why I consider using PySide instead of
> PyQt4.
>
> Thanks for reading!
>
> Cheers,
>
> Jean-Louis

Hi Jean-Louis,

On Thu, Oct 13, 2011 at 1:47 AM, Durrieu Jean-Louis
<jea...@ep...> wrote:
>
> Is that the right way to do so? Is there any way of setting the QT_API more "definitely", and not having to change the call to python?
>

Yes, in your $HOME/.bashrc file that your shell loads at startup time,
you can write

export QT_API="pyside"

And this will ensure that you always have this variable set as part of
your 'environment'.  The concept of 'environment' is a unix-specific
idea of a set of variables and other parameters visible to any
program; you can actually read and write the enviroment in python by
importing 'os' and using the os.environ variable:

$ export SOMEVAR="Hello matplotlib"
$ ipython --no-banner

In [1]: import os

In [2]: os.environ['SOMEVAR']
Out[2]: 'Hello matplotlib'

Cheers,

f

I use matplotlib 1.1 and numpy 1.5.0

gc.collect() after the function call solved the problem! Thanks alot!

Den torsdagen den 13:e oktober 2011 skrev John Hunter<jd...@gm...>:
> On Thu, Oct 13, 2011 at 12:42 AM, Viktor Forsman
> <vik...@gm...> wrote:
>> Hi,
>>
>> I have problems with a memory leak in a webapplication involving
matplotlib.
>> Basically, I have a function which usies the OO way of generating the
graph,
>> printing it to a StringIO object and returning that. I've posted a
question
>> with the details on stackoverflow
>> (
http://stackoverflow.com/questions/7740832/need-help-identifying-a-memory-leak-involving-matplotlib-and-flask
).
>> Any ideas are highly appreciated.
>
> What version of matplotlib and numpy are you using?  Does it help to do:
>
>    import gc
>    gc.collect()
>
> after a call to plot total or plot month?
>
> Also, you appear to be adding data to "plot_values" and storing this
> on every call.  Are you sure the leak is in mpl and not here?
> JDH
>

Hi,

Not directly answering your questions but the code below produces what you
are trying to achieve:

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid.parasite_axes import SubplotHost

plt.close('all')
fig1 = plt.figure(figsize=(11, 8.5))
ax1 = SubplotHost(fig1, 111)
fig1.add_subplot(ax1)

## define data
x = np.arange(1,11)
y = np.random.random(len(x))

# First X-axis
ax1.plot(x, y)
xmin, xmax = ax1.xaxis.get_view_interval()
ax1.xaxis.set_label_text("First X-axis")
ax1.yaxis.set_label_text("Sample data")

# Second X-axis
parx = ax1.twiny()
offset = 0, -60
new_axisline = parx.get_grid_helper().new_fixed_axis
parx.axis["bottom"] = new_axisline(loc="bottom", axes=parx, offset=offset)
parx.axis["bottom"].label.set_text("Second X-axis")
line1, = parx.plot((1./x), np.ones(len(x)))
line1.set_visible(0)
parx.set_xlim(xmin=xmax,xmax=xmin)
parx.axis["top"].set_visible(False)

ax1.grid(1)
plt.tight_layout()
plt.show()


A few notes:

1-) axes is plural for axis. I always get confused with these words. Tricky
English :)

2-) This code uses the older AxisGrid toolkit functions, and not super
efficient since it makes an empty plot call with a different x-units.
However, using the same ideas and with some dedication you can create
multi-axes like the one shown here -
http://imageshack.us/photo/my-images/820/plot1r.png/

I had tried the newer AxisGrid approach as JJ suggested but I failed making
the plot look pretty. Plus sometimes using the AffineTransforms are not very
practical.

3-) I said "empty plot call" but it is actually where you convert your unit.
(np.ones call is another dummy call since the line is made invisible after
its call) To me it is more explicit than converting units via transforms. In
this example I just assume x as wavelength and convert it to wavenumber with
1/x. You need to adjust that call according to your unit conversion as well
as adjusting the limits of your newly scaled unit.

As usual JJ can give a more elegant solution for your question, but
sometimes practicality beats purity and you engineer your own solution.

On Wed, Oct 12, 2011 at 11:36 PM, List Comprehension <
lis...@gm...> wrote:

>
> Hello, I am interested in plotting two (x) axes for the same figure (one of
> is just showing linearly transformed values of the other); one of them
> offset from the plot if possible. For instance, I have the following code:
>
> import numpy as np
> import matplotlib.pyplot as plt
> import mpl_toolkits.axisartist as AA
>
> ## define data
> x = np.array(range(11))
> xlab = x*2
> y = np.random.random(len(x))
>
> fig = plt.figure()
> ## first axes which is only for drawing the axis
> axsecond = AA.Subplot(fig, 1,1,1)
> fig.add_subplot(axsecond)
> axsecond.set_xlim([min(xlab),max(xlab)])
> axsecond.set_xticks(np.linspace(min(xlab),max(xlab),5))
> axsecond.axis["bottom", "top", "left", "right"].set_visible(False)
> axsecond.axis["newx"] = axsecond.new_floating_axis(nth_coord=0, value=1)
> axsecond.axis["newx"].set_axis_direction("top")
> axsecond.set_clip_on(False)
> axsecond.axis["newx"].toggle(all=True)
> axsecond.axis["newx"].set_visible(True)
> plt.draw()
> ## the main set of axes containing the data
> axmain = fig.add_subplot(1,1,1)
> plt.plot(x,y)
>
>
> In this example I wonder
>
> 1) How to turn off the top axis of axmain so the ticks from axsecond are
> visible? Something like axmain["top"].set_visible(False) (which does not
> work of course).
>
> 2) If I wanted to include an offset (below the main axis), I feel like
> something like the following should work but doesn't:
>
> ax.axis["newx"] = ax.new_floating_axis(nth_coord=0, value=-0.2) ## outside
> of (0,1)
> ax.axis["newx"].set_axis_direction("bottom")
> ax.set_clip_on(False)
>
> Thanks in advance!
>



-- 
Gökhan

On Thursday, October 13, 2011, Neal Becker <ndb...@gm...> wrote:
> Using interactively (via emacs/ipython), on closing a plot window I see:
>
>  X Error: BadWindow (invalid Window parameter) 3
>  Major opcode: 20 (X_GetProperty)
>  Resource id:  0x5802e1b
>
>

Which backend/platform?

Ben Root

linux fedora 15 x86_64

backend : Qt4Agg

Now it's gone away, after killing the *Python* buffer and restarting the python 
process.  If it comes back I'll try to get more info.


John Hunter wrote:

> On Thu, Oct 13, 2011 at 8:03 AM, Neal Becker
> <ndb...@gm...> wrote:
>> Using interactively (via emacs/ipython), on closing a plot window I see:
>>
>> X Error: BadWindow (invalid Window parameter) 3
>> Major opcode: 20 (X_GetProperty)
>> Resource id:  0x5802e1b
> 
> Could you give us some more information.
> 
> What operating system?
> 
> Which matplotlib backend and GUI version?
> 
> What version of ipython?
> 
> What, exactly do you do to reproduce the problem?
> 
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct

On Thu, Oct 13, 2011 at 8:03 AM, Neal Becker <ndb...@gm...> wrote:
> Using interactively (via emacs/ipython), on closing a plot window I see:
>
>  X Error: BadWindow (invalid Window parameter) 3
>  Major opcode: 20 (X_GetProperty)
>  Resource id:  0x5802e1b

Could you give us some more information.

What operating system?

Which matplotlib backend and GUI version?

What version of ipython?

What, exactly do you do to reproduce the problem?

Using interactively (via emacs/ipython), on closing a plot window I see:

 X Error: BadWindow (invalid Window parameter) 3
  Major opcode: 20 (X_GetProperty)
  Resource id:  0x5802e1b

On Thu, Oct 13, 2011 at 12:42 AM, Viktor Forsman
<vik...@gm...> wrote:
> Hi,
>
> I have problems with a memory leak in a webapplication involving matplotlib.
> Basically, I have a function which usies the OO way of generating the graph,
> printing it to a StringIO object and returning that. I've posted a question
> with the details on stackoverflow
> (http://stackoverflow.com/questions/7740832/need-help-identifying-a-memory-leak-involving-matplotlib-and-flask).
> Any ideas are highly appreciated.

What version of matplotlib and numpy are you using?  Does it help to do:

    import gc
    gc.collect()

after a call to plot total or plot month?

Also, you appear to be adding data to "plot_values" and storing this
on every call.  Are you sure the leak is in mpl and not here?
JDH

Dear all, matplotlib users and devs,

I have been trying the latest announced version of Matplotlib (1.1.0). The new feature that attracted me is the PySide support. 

However, I tried to install it under Ubuntu 11.04, but I needed pyqt4 installed in order to install it from source. Is there a way to install it with only pyside installed? 

Once installed, I could uninstall pyqt4, and run my python scripts, with replacing pyqt4 by pyside. However, I am not sure what the "ETS environment variable" is (yes, it s related to the enthought tool suite, but what if I do not want it - for now), and I had to set the QT_API variable on the command line call to python: setting it as a "normal" environment variable does not work. So, to be able to use "only" pyside, I need to do:

QT_API=pyside python myscript.py

Is that the right way to do so? Is there any way of setting the QT_API more "definitely", and not having to change the call to python?

So far, my program (I really have only one at the moment) seems to work well, using QtCore, QtGui and phonon, except for phonon, where it seems that "seeking" in a file does not really work under Ubuntu... if anyone has an idea (I know, not the right mailing list, but just in case).

At last, probably a question that will solve all the above ones: when is the new Matplotlib version going to be included in EPD? That's, by the way, mainly the reason why I consider using PySide instead of PyQt4. 

Thanks for reading!

Cheers,

Jean-Louis

Hi,

I have problems with a memory leak in a webapplication involving matplotlib.
Basically, I have a function which usies the OO way of generating the graph,
printing it to a StringIO object and returning that. I've posted a question
with the details on stackoverflow (
http://stackoverflow.com/questions/7740832/need-help-identifying-a-memory-leak-involving-matplotlib-and-flask).
Any ideas are highly appreciated.

Hello, I am interested in plotting two (x) axes for the same figure (one of
is just showing linearly transformed values of the other); one of them
offset from the plot if possible. For instance, I have the following code:

import numpy as np
import matplotlib.pyplot as plt
import mpl_toolkits.axisartist as AA

## define data
x = np.array(range(11))
xlab = x*2
y = np.random.random(len(x))

fig = plt.figure()
## first axes which is only for drawing the axis
axsecond = AA.Subplot(fig, 1,1,1)
fig.add_subplot(axsecond)
axsecond.set_xlim([min(xlab),max(xlab)])
axsecond.set_xticks(np.linspace(min(xlab),max(xlab),5))
axsecond.axis["bottom", "top", "left", "right"].set_visible(False)
axsecond.axis["newx"] = axsecond.new_floating_axis(nth_coord=0, value=1)
axsecond.axis["newx"].set_axis_direction("top")
axsecond.set_clip_on(False)
axsecond.axis["newx"].toggle(all=True)
axsecond.axis["newx"].set_visible(True)
plt.draw()
## the main set of axes containing the data
axmain = fig.add_subplot(1,1,1)
plt.plot(x,y)


In this example I wonder

1) How to turn off the top axis of axmain so the ticks from axsecond are
visible? Something like axmain["top"].set_visible(False) (which does not
work of course).

2) If I wanted to include an offset (below the main axis), I feel like
something like the following should work but doesn't:

ax.axis["newx"] = ax.new_floating_axis(nth_coord=0, value=-0.2) ## outside
of (0,1)
ax.axis["newx"].set_axis_direction("bottom")
ax.set_clip_on(False)

Thanks in advance!

Hi All,
I keep receiving a memory error when processing many netcdf files. I assumed
it had something to do with how I loop things and maybe needed to close
things off properly but I recently received an error that made me think it
might be because of matplotlib.

In the code below I am looping through a bunch of netcdf files (each file is
hourly data for one month) and within each netcdf file I am outputting a
*png file every three hours. This works for one netcdf file (therefore one
month) but when it begins to process the next netcdf file I receive a memory
error (see below). Since I have tidied some of my code up it seems to
process partly into the second file but then I still receive the memory
error.
I have tried a few suggestions such as:
-Combining the dataset using MFDataset (using NETCDF4) is not an option
because the files do not have unlimited dimension.
- gc.collect() but that just results in a *GEOS_ERROR: bad allocation error*.

-only open LAT and LON once (which worked)

System Details:
Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC
v.1500 32 bit (Intel)] on win32

Any feedback will be greatly appreciated as I seem to keep ending up with
memory errors when working with netcdf files this even happens if I am using
a much better computer.

*Most recent error: *
Traceback (most recent call last):
  File "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line 78,
in <module>
    plt.savefig((os.path.join(outputfolder, 'TSFC'+date_string+'UTC.png')))
  File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line 363, in
savefig
    return fig.savefig(*args, **kwargs)
  File "C:\Python27\lib\site-packages\matplotlib\figure.py", line 1084, in
savefig
    self.canvas.print_figure(*args, **kwargs)
  File "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py",
line 100, in print_figure
    FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
  File "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
1923, in print_figure
    **kwargs)
  File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
line 438, in print_png
    FigureCanvasAgg.draw(self)
  File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
line 393, in draw
    self.renderer = self.get_renderer()
  File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
line 404, in get_renderer
    self.renderer = RendererAgg(w, h, self.figure.dpi)
  File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
line 59, in __init__
    self._renderer = _RendererAgg(int(width), int(height), dpi, debug=False)
RuntimeError: Could not allocate memory for image

*Error when I added gc.collect()*
GEOS_ERROR: bad allocation

*Old error (before adding gc.collect() )*
*Traceback (most recent call last):
  File "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
line 44, in <module>
    TSFC=ncfile.variables['T_SFC'][1::3]
  File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
(netCDF4.c:23094)
MemoryError*



from netCDF4 import Dataset
import numpy as N
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
from netcdftime import utime
from datetime import datetime
import os
import gc


shapefile1="E:/
>
> griddeddatasamples/GIS/DSE_REGIONS"
> MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
> OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
> fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
> 'r+', 'NETCDF4')
> LAT=fileforlatlon.variables['latitude'][:]
> LON=fileforlatlon.variables['longitude'][:]
>
> for (path, dirs, files) in os.walk(MainFolder):
>     for dir in dirs:
>         print dir
>     path=path+'/'
>     for ncfile in files:
>         if ncfile[-3:]=='.nc':
>             print "dealing with ncfiles:", ncfile
>             ncfile=os.path.join(path,ncfile)
>             ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>             TSFC=ncfile.variables['T_SFC'][1::3]
>             TIME=ncfile.variables['time'][1::3]
>             ncfile.close()
>             gc.collect()
>
>             for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
>                 cdftime=utime('seconds since 1970-01-01 00:00:00')
>                 ncfiletime=cdftime.num2date(TIME)
>                 print ncfiletime
>                 timestr=str(ncfiletime)
>                 d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
>                 date_string = d.strftime('%Y%m%d_%H%M')
>
>                 map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>                 x,y=map(*N.meshgrid(LON,LAT))
>                 map.drawcoastlines(linewidth=0.5)
>                 map.readshapefile(shapefile1, 'DSE_REGIONS')
>                 map.drawstates()
>
>                 plt.title('Surface temperature at %s UTC'%ncfiletime)
>                 ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
>                 CS = map.contourf(x,y,TSFC, ticks, cmap=plt.cm.jet)
>                 l,b,w,h =0.1,0.1,0.8,0.8
>                 cax = plt.axes([l+w+0.025, b, 0.025, h], )
>                 cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>
>                 plt.savefig((os.path.join(OutputFolder,
> 'TSFC'+date_string+'UTC.png')))
>                 plt.close()
>                 gc.collect()
>
>
>

Matplotlib 1.1 fixes the NavigationToolbar2Wx issue with saving the .png image.

Thanks for the excellent support.

Regards.
Keith

________________________________
From: ben...@gm... [mailto:ben...@gm...] On Behalf Of Benjamin Root
Sent: Saturday, 24 September 2011 2:59 a.m.
To: Keith Jones
Cc: mat...@li...
Subject: Re: [Matplotlib-users] NavigationToolbar2Wx with mplot3d



On Wednesday, September 21, 2011, Benjamin Root <ben...@ou...<mailto:ben...@ou...>> wrote:
>
>
> On Wednesday, September 21, 2011, Keith Jones <K....@ir...<mailto:K....@ir...>> wrote:
>> Hi,
>>
>> I have two questions about using NavigationToolbar2Wx with mplot3d.
>>
>>
>>
>> 1/  Initially the 3D scatter plot will rotate as usual with a mouse, but after selecting the 'pan' or 'zoom' buttons the plot responds with some confusion.  How can I restore it to rotation only, i.e. disconnect the zoom or pan behaviour?
>>
>
> That is a bug that should be resolved in the upcoming release.  Use the right mouse button for zooming instead.  In the upcoming release, the zoom and pan button should have no effect on axes3d objects, if I remember correctly.
>
>
>>
>>
>> 2/  When using the 'save' button I get different behaviours depending on the backend.  With 'WXAgg' the saved png image shows only the axes, not the scatter points.  The scatter points and axes do appear correctly in a pdf file.  Using the 'WX' backend gives both scatter points and axes for the png file.
>>
>
> I dont use WxAgg regularly.  I will use your code to test this.
>
> Ben Root

I could not reproduce your problem with WxAgg backend on the development branch.  I could only test on Linux, though. However, if there is a difference between platforms with WxAgg backend, then it is likely a Wx bug and not a mpl bug (although the fact that the Wx backend worked is odd).

We are putting out an RC of v1.1.0 later today. Could you try out that version within the next few days and let me know if it still happens for you?

Ben Root

________________________________
This electronic transmission and any documents accompanying this electronic transmission contain confidential information belonging to the sender. This information may be legally privileged. The information is intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited.