matplotlib-users Mailing List for matplotlib

It looks like you are calling `pcolor`.  Can I suggest you try `pcolormesh`?  

75 Mb is not a big file!

Cheers,   Jody


On Sep 8, 2014, at  7:38 AM, Benjamin Root <ben...@ou...> wrote:

> (Keeping this on the mailing list so that others can benefit)
> 
> What might be happening is that you are keeping around too many numpy arrays in memory than you actually need. Take advantage of memmapping, which most netcdf tools provide by default. This keeps the data on disk rather than in RAM. Second, for very large images, I would suggest either pcolormesh() or just simply imshow() instead of pcolor() as they are more way more efficient than pcolor(). In addition, it sounds like you are dealing with re-sampled data ("at different zoom levels"). Does this mean that you are re-running contour on re-sampled data? I am not sure what the benefit of doing that is if one could just simply do the contour once at the highest resolution.
> 
> Without seeing any code, though, I can only provide generic suggestions.
> 
> Cheers!
> Ben Root
> 
> 
> On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <raf...@li...> wrote:
> Hi Ben,
> 
> sorry for the few details that I gave to you. I'm trying to make a contour plot of a variable at different zoom levels by using high resolution data. The aim is to obtain .PNG output images. Actually, I'm working with big data (NetCDF file, dimension is about 75Mb). The current Matplotlib version on my UBUNTU 14.04 machine is the 1.3.1 one. My system has a RAM capacity of 8Gb.
> Actually, I'm dealing with memory system problems when I try to make a plot. I got the error message as follow:
> 
> --------------------------------------------
>      cs = m.pcolor(xi,yi,np.squeeze(t))
>   File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line 521, in with_transform
>     return plotfunc(self,x,y,data,*args,**kwargs)
>   File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line 3375, in pcolor
>     x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
>   File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195, in masked_values
>     condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
> MemoryError  
> --------------------------------------------
> 
> Otherwise, when I try to make a plot of smaller file (such as 5Mb), it works very well. I believe that it's not something of wrong in the script. It might be a memory system problem.
> I hope that my message is more clear now.
> 
> Thanks for the help.
> 
> Regards,
> 
> Raffaele
> 
> -----------------------------------------
> 
> Sent: Mon 9/8/2014 3:19 PM
> To: Raffaele Quarta
> Cc: Matplotlib Users
> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
> 
> 
> 
> You will need to be more specific... much more specific. What kind of plot
> are you making? How big is your data? What version of matplotlib are you
> using? How much RAM do you have available compared to the amount of data
> (most slowdowns are actually due to swap-thrashing issues). Matplotlib can
> be used for large data, but there exists some speciality tools for the
> truly large datasets. The solution depends on the situation.
> 
> Ben Root
> 
> On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <raf...@li...>
> wrote:
> 
> >  Hi,
> >
> > I'm working with NetCDF format. When I try to make a plot of very large
> > file, I have to wait for a long time for plotting. How can I solve this?
> > Isn't there a solution for this problem?
> >
> > Raffaele
> >
> > --
> > This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> > http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
> 
> --
> This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
> 
> 
> 
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

--
Jody Klymak    
http://web.uvic.ca/~jklymak/

(Keeping this on the mailing list so that others can benefit)

What might be happening is that you are keeping around too many numpy
arrays in memory than you actually need. Take advantage of memmapping,
which most netcdf tools provide by default. This keeps the data on disk
rather than in RAM. Second, for very large images, I would suggest either
pcolormesh() or just simply imshow() instead of pcolor() as they are more
way more efficient than pcolor(). In addition, it sounds like you are
dealing with re-sampled data ("at different zoom levels"). Does this mean
that you are re-running contour on re-sampled data? I am not sure what the
benefit of doing that is if one could just simply do the contour once at
the highest resolution.

Without seeing any code, though, I can only provide generic suggestions.

Cheers!
Ben Root


On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <raf...@li...
> wrote:

>  Hi Ben,
>
> sorry for the few details that I gave to you. I'm trying to make a contour
> plot of a variable at different zoom levels by using high resolution data.
> The aim is to obtain .PNG output images. Actually, I'm working with big
> data (NetCDF file, dimension is about 75Mb). The current Matplotlib version
> on my UBUNTU 14.04 machine is the 1.3.1 one. My system has a RAM capacity
> of 8Gb.
> Actually, I'm dealing with memory system problems when I try to make a
> plot. I got the error message as follow:
>
> --------------------------------------------
>      cs = m.pcolor(xi,yi,np.squeeze(t))
>   File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
> line 521, in with_transform
>     return plotfunc(self,x,y,data,*args,**kwargs)
>   File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
> line 3375, in pcolor
>     x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
>   File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195, in
> masked_values
>     condition = umath.less_equal(mabs(xnew - value), atol + rtol *
> mabs(value))
> MemoryError
> --------------------------------------------
>
> Otherwise, when I try to make a plot of smaller file (such as 5Mb), it
> works very well. I believe that it's not something of wrong in the script.
> It might be a memory system problem.
> I hope that my message is more clear now.
>
> Thanks for the help.
>
> Regards,
>
> Raffaele
>
> -----------------------------------------
>
> Sent: Mon 9/8/2014 3:19 PM
> To: Raffaele Quarta
> Cc: Matplotlib Users
> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>
>
> You will need to be more specific... much more specific. What kind of plot
> are you making? How big is your data? What version of matplotlib are you
> using? How much RAM do you have available compared to the amount of data
> (most slowdowns are actually due to swap-thrashing issues). Matplotlib can
> be used for large data, but there exists some speciality tools for the
> truly large datasets. The solution depends on the situation.
>
> Ben Root
>
> On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <
> raf...@li...>
> wrote:
>
> >  Hi,
> >
> > I'm working with NetCDF format. When I try to make a plot of very large
> > file, I have to wait for a long time for plotting. How can I solve this?
> > Isn't there a solution for this problem?
> >
> > Raffaele
> >
> > --
> > This email was Virus checked by Astaro Security Gateway.
> http://www.sophos.com
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
>
> --
> This email was Virus checked by Astaro Security Gateway.
> http://www.sophos.com
>
>

You will need to be more specific... much more specific. What kind of plot
are you making? How big is your data? What version of matplotlib are you
using? How much RAM do you have available compared to the amount of data
(most slowdowns are actually due to swap-thrashing issues). Matplotlib can
be used for large data, but there exists some speciality tools for the
truly large datasets. The solution depends on the situation.

Ben Root

On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <raf...@li...>
wrote:

>  Hi,
>
> I'm working with NetCDF format. When I try to make a plot of very large
> file, I have to wait for a long time for plotting. How can I solve this?
> Isn't there a solution for this problem?
>
> Raffaele
>
> --
> This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Hi,

I'm working with NetCDF format. When I try to make a plot of very large file, I have to wait for a long time for plotting. How can I solve this? Isn't there a solution for this problem? 

Raffaele  

-- 
This email was Virus checked by Astaro Security Gateway. http://www.sophos.com

Le 05/09/2014 21:53, Arnaldo Russo a écrit :
> The following code plots my table, but greek letters are not in Arial.
What about adding greek letters directly with a Unicode string and 
keeping LaTex only for the table?

best,
Pierre

(my greek and math unicode "copy-pasting files" attached)

Hi,

I'm having trouble with Basemap South Pole Stereographic
projections. The required "boundinglat" doesn't seem to work properly,
and I hope someone can help me figure out the correct syntax.

I have the following information about the data:

extent: -3,333,500 to 3,333,500 m
true scale latitude: -71
south pole stereographic projection.

Given this, I try the following:

m = Basemap(resolution='c',projection='spstere',
lat_ts=-71,lat_0=0.,lon_0=0.)

And it crashes reporting that I need boundinglat. I can't give
width/height which are accepted by the 'stere' projection.

I use http://www.pgc.umn.edu/tools/conversion and determine that
3333500 is 60 degrees south. If then add "boundinglat=-60" to the
Basemap call, the data appears nowhere near the correct location.

If I add "boundinglat=-89.999999" it looks almost perfect. If I change
that to "-90" or add a few more 9's, it crashes.

Am I determining the bounding lat correctly from the provided
information? Should I be calling Basemap differently? Any help will be
much appreciated.

Thanks,

  Ken Mankoff