matplotlib-users Mailing List for matplotlib

Thanks Jeff!

On Thu, Oct 20, 2011 at 1:16 PM, Jeff Whitaker <js...@fa...> wrote:

>  On 10/19/11 4:37 PM, questions anon wrote:
>
> thank you, I am not quite sure how to 'draw' the shapefile
>
>             from matplotlib.collections import LineCollection
>             ax = plt.gca() # get current axes instance
> # 'DSE_REGIONS' instance variable created by readshapefile method call.
>             lines = LineCollection(map.DSE_REGIONS)
>             ax.add_collection(lines)
>
> -Jeff
>
>
>  but making those changes and removing the shapefile has sped the
> processing up considerably!
> Thank you for your help
>
> On Wed, Oct 19, 2011 at 11:42 PM, Jeff Whitaker <js...@fa...>wrote:
>
>>  On 10/18/11 8:55 PM, questions anon wrote:
>>
>> Thanks Jeff, that certainly speeds it up! But when I take them out of the
>> loop and place them elsewhere they are no longer added to the map.
>> Is there someway I can call them in the loop but still get it to run
>> quickly?
>> Thanks
>>
>>
>>  Just the Basemap instance creation and the transformation of coordinates
>> to projection space should be hoisted out of the loop
>>
>>
>>                 map =
>> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>>
>> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>>                 x,y=map(*N.meshgrid(LON,LAT))
>>
>>  you can leave the other statements in.
>>
>> If you still have memory issues, bring the readshapefile call out, and
>> draw the shapes whose coordinates are stored in the instance variable
>> map.DSE_REGIONS manually in the loop.
>>
>> -Jeff
>>
>>
>> On Fri, Oct 14, 2011 at 10:54 PM, Jeff Whitaker <js...@fa...>wrote:
>>
>>   On 10/12/11 8:20 PM, questions anon wrote:
>>
>> Hi All,
>> I keep receiving a memory error when processing many netcdf files. I
>> assumed it had something to do with how I loop things and maybe needed to
>> close things off properly but I recently received an error that made me
>> think it might be because of matplotlib.
>>
>> In the code below I am looping through a bunch of netcdf files (each file
>> is hourly data for one month) and within each netcdf file I am outputting a
>> *png file every three hours. This works for one netcdf file (therefore one
>> month) but when it begins to process the next netcdf file I receive a memory
>> error (see below). Since I have tidied some of my code up it seems to
>> process partly into the second file but then I still receive the memory
>> error.
>> I have tried a few suggestions such as:
>> -Combining the dataset using MFDataset (using NETCDF4) is not an option
>> because the files do not have unlimited dimension.
>> - gc.collect() but that just results in a *GEOS_ERROR: bad allocation
>> error*.
>> -only open LAT and LON once (which worked)
>>
>> System Details:
>> Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC
>> v.1500 32 bit (Intel)] on win32
>>
>> Any feedback will be greatly appreciated as I seem to keep ending up with
>> memory errors when working with netcdf files this even happens if I am using
>> a much better computer.
>>
>> *Most recent error: *
>> Traceback (most recent call last):
>>   File "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line
>> 78, in <module>
>>     plt.savefig((os.path.join(outputfolder,
>> 'TSFC'+date_string+'UTC.png')))
>>   File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line 363, in
>> savefig
>>     return fig.savefig(*args, **kwargs)
>>   File "C:\Python27\lib\site-packages\matplotlib\figure.py", line 1084, in
>> savefig
>>     self.canvas.print_figure(*args, **kwargs)
>>   File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py", line
>> 100, in print_figure
>>     FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
>>   File "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
>> 1923, in print_figure
>>     **kwargs)
>>   File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 438, in print_png
>>     FigureCanvasAgg.draw(self)
>>   File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 393, in draw
>>     self.renderer = self.get_renderer()
>>   File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 404, in get_renderer
>>     self.renderer = RendererAgg(w, h, self.figure.dpi)
>>   File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 59, in __init__
>>     self._renderer = _RendererAgg(int(width), int(height), dpi,
>> debug=False)
>> RuntimeError: Could not allocate memory for image
>>
>> *Error when I added gc.collect()*
>> GEOS_ERROR: bad allocation
>>
>> *Old error (before adding gc.collect() )*
>> *Traceback (most recent call last):
>>   File
>> "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py", line
>> 44, in <module>
>>     TSFC=ncfile.variables['T_SFC'][1::3]
>>   File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
>> (netCDF4.c:23094)
>> MemoryError*
>>
>>
>>
>> from netCDF4 import Dataset
>> import numpy as N
>> import matplotlib.pyplot as plt
>> from mpl_toolkits.basemap import Basemap
>> from netcdftime import utime
>> from datetime import datetime
>> import os
>> import gc
>>
>>
>> shapefile1="E:/
>>
>> griddeddatasamples/GIS/DSE_REGIONS"
>> MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
>> OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
>> fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
>> 'r+', 'NETCDF4')
>> LAT=fileforlatlon.variables['latitude'][:]
>> LON=fileforlatlon.variables['longitude'][:]
>>
>> for (path, dirs, files) in os.walk(MainFolder):
>>     for dir in dirs:
>>         print dir
>>     path=path+'/'
>>     for ncfile in files:
>>         if ncfile[-3:]=='.nc':
>>             print "dealing with ncfiles:", ncfile
>>             ncfile=os.path.join(path,ncfile)
>>             ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>>             TSFC=ncfile.variables['T_SFC'][1::3]
>>             TIME=ncfile.variables['time'][1::3]
>>             ncfile.close()
>>             gc.collect()
>>
>>             for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
>>                 cdftime=utime('seconds since 1970-01-01 00:00:00')
>>                 ncfiletime=cdftime.num2date(TIME)
>>                 print ncfiletime
>>                 timestr=str(ncfiletime)
>>                 d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
>>                 date_string = d.strftime('%Y%m%d_%H%M')
>>
>>                 map =
>> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>>
>> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>>                 x,y=map(*N.meshgrid(LON,LAT))
>>                 map.drawcoastlines(linewidth=0.5)
>>                 map.readshapefile(shapefile1, 'DSE_REGIONS')
>>                 map.drawstates()
>>
>>                 plt.title('Surface temperature at %s UTC'%ncfiletime)
>>                 ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
>>                 CS = map.contourf(x,y,TSFC, ticks, cmap=plt.cm.jet)
>>                 l,b,w,h =0.1,0.1,0.8,0.8
>>                 cax = plt.axes([l+w+0.025, b, 0.025, h], )
>>                 cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>>
>>                 plt.savefig((os.path.join(OutputFolder,
>> 'TSFC'+date_string+'UTC.png')))
>>                 plt.close()
>>                 gc.collect()
>>
>>
>>  Try moving these lines
>>
>>
>>                 map =
>> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>>
>> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>>                 x,y=map(*N.meshgrid(LON,LAT))
>>                 map.drawcoastlines(linewidth=0.5)
>>                 map.readshapefile(shapefile1, 'DSE_REGIONS')
>>                 map.drawstates()
>>
>>  out of the loop.
>>
>> -Jeff
>>
>>
>>
>>
>
>

Would it possible to update the matplotlib page on pypi.python.org to 1.1?
Otherwise, by default easy_install and pip currently install 1.0.1.

Thank you,
--
Félix-Antoine Fortin

On Sat, Oct 22, 2011 at 7:54 AM, Rich Shepard <rsh...@ap...> wrote:
> On Fri, 21 Oct 2011, Paul Ivanov wrote:
>   I will certainly add diagnostic code requested by you, Ben, and anyone
> else and report the results when trying to run the model. I do need to fix
> this and have no idea what's behind the problem.

The traceback is due to a nonprinting character being included in one
of the fonts on your system - we just need to figure out which one.
Change my request to add 'sys.stderr.write(fh.name)' before the 'while
1:' in
_parse_char_metrics  - just so we don't have any buffering issues. The
last file you see printed there will be the one that's causing the
issue. You can then try removing it, or sending it back to the list
(or both) so we can see what happened.

The other issue you're seeing ("unknown keyword in AFM header") is
also likely caused by bad font files.
>From your error log that Ben forwarded to the list - I'm a bit
suspicious that two of the errors came from an afm file that ships
with matplotlib, in particular matplotlib-error-trace.txt starts off
with:

  FILENAME:
  /usr/lib/python2.6/site-packages/matplotlib/mpl-data/fonts/afm/pagko8a.afm
  Found an unknown keyword in AFM header (was Underline)
  Found an unknown keyword in AFM header (was Underline)

This shouldn't be the case, as I can verify that the keywords aren't
just Underline - they are as follows:

  $ grep Under matplotlib/mpl-data/fonts/afm/pagko8a.afm
  UnderlinePosition -100
  UnderlineThickness 50

and that particular file has not been change in matplotlib since
February of 2007

So my wild guess is that something changed your afm files. Can you
check that your pagko8a.afm matches the one in
https://raw.github.com/matplotlib/matplotlib/master/lib/matplotlib/mpl-data/fonts/afm/pagko8a.afm
- and if they don't match, check that using the original pagko8a.afm
makes at least that error go away?

As another possible solution (maybe this is already done, but worth a
potshot), you could try switching USE_FONTCONFIG to True in
font_manager.py as per the docstring there:

Experimental support is included for using `fontconfig` on Unix
variant platforms (Linux, OS X, Solaris).  To enable it, set the
constant ``USE_FONTCONFIG`` in this file to ``True``.  Fontconfig has
the advantage that it is the standard way to look up fonts on X11
platforms, so if a font is installed, it is much more likely to be
found.)

best,
-- 
Paul Ivanov
314 address only used for lists,  off-list direct email at:
http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7

Yarr.  Sorry for the noise.  Just needed to install the freetype headers :|

apt-get install libfreetype6-dev

--
Jake Biesinger
Graduate Student
Xie Lab, UC Irvine



On Sat, Oct 22, 2011 at 11:40 PM, Jacob Biesinger
<jak...@gm...>wrote:

> Hi!
>
> Trying to upgrade my matplotlib to use the new 3d plotting tools.
>
> $ sudo pip install -U matplotlib
> ...
> building 'matplotlib.ft2font' extension
>
> creating build/temp.linux-x86_64-2.7
>
> creating build/temp.linux-x86_64-2.7/src
>
> creating build/temp.linux-x86_64-2.7/CXX
>
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MP
> L_ARRAY_API -DPYCXX_ISO_CPP_LIB=1
> -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/local/include
> -I/usr/inclu
> de -I. -I/usr/lib/pymodules/python2.7/numpy/core/include/freetype2
> -I/usr/local/include/freetype2 -I/usr/include/free
> type2 -I./freetype2 -I/usr/include/python2.7 -c src/ft2font.cpp -o
> build/temp.linux-x86_64-2.7/src/ft2font.o
>
> cc1plus: warning: command line option "-Wstrict-prototypes" is valid for
> Ada/C/ObjC but not for C++
>
> In file included from src/ft2font.cpp:1:0:
>
> src/ft2font.h:14:22: fatal error: ft2build.h: No such file or director
>
>
> and it looks like the file really is missing...
> $ find build/ -name *ft2build*
> $ ls build/matplotlib/src/
> agg_py_path_iterator.h  _backend_gdk.c  _gtkagg.cpp  mplutils.cpp
>  path_cleanup.h     _tkagg.cpp
> agg_py_transforms.cpp   backend_gdk.c   _image.cpp   mplutils.h
>  path_converters.h  _ttconv.cpp
> agg_py_transforms.h     cntr.c          _image.h     numerix.h
> _path.cpp          _windowing.cpp
> _backend_agg.cpp        ft2font.cpp     _macosx.m    nxutils.c
> _png.cpp           _wxagg.cpp
> _backend_agg.h          ft2font.h       MPL_isnan.h  path_cleanup.cpp
>  _subprocess.c
>
>
> Perhaps someone forgot to hg add a file?
>
> --
> Jake Biesinger
> Graduate Student
> Xie Lab, UC Irvine
>
>

Hi!

Trying to upgrade my matplotlib to use the new 3d plotting tools.

$ sudo pip install -U matplotlib
...
building 'matplotlib.ft2font' extension

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/src

creating build/temp.linux-x86_64-2.7/CXX

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MP
L_ARRAY_API -DPYCXX_ISO_CPP_LIB=1
-I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/local/include
-I/usr/inclu
de -I. -I/usr/lib/pymodules/python2.7/numpy/core/include/freetype2
-I/usr/local/include/freetype2 -I/usr/include/free
type2 -I./freetype2 -I/usr/include/python2.7 -c src/ft2font.cpp -o
build/temp.linux-x86_64-2.7/src/ft2font.o

cc1plus: warning: command line option "-Wstrict-prototypes" is valid for
Ada/C/ObjC but not for C++

In file included from src/ft2font.cpp:1:0:

src/ft2font.h:14:22: fatal error: ft2build.h: No such file or director


and it looks like the file really is missing...
$ find build/ -name *ft2build*
$ ls build/matplotlib/src/
agg_py_path_iterator.h  _backend_gdk.c  _gtkagg.cpp  mplutils.cpp
 path_cleanup.h     _tkagg.cpp
agg_py_transforms.cpp   backend_gdk.c   _image.cpp   mplutils.h
 path_converters.h  _ttconv.cpp
agg_py_transforms.h     cntr.c          _image.h     numerix.h
_path.cpp          _windowing.cpp
_backend_agg.cpp        ft2font.cpp     _macosx.m    nxutils.c
_png.cpp           _wxagg.cpp
_backend_agg.h          ft2font.h       MPL_isnan.h  path_cleanup.cpp
 _subprocess.c


Perhaps someone forgot to hg add a file?

--
Jake Biesinger
Graduate Student
Xie Lab, UC Irvine