matplotlib-users Mailing List for matplotlib

On Mon, May 6, 2013 at 3:53 PM, Michael Droettboom <md...@st...> wrote:
> My understanding is that distutils builds up the commandline arguments for
> gcc in this order:
>
>   1) From Python's Makefile.
>   2) From environment variables
>   3) From whatever was added by the setup.py script
>
> It looks like you have some extra stuff in (1), namely
>
> -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu
>
> You can find the Python Makefile that is being used to source this
> information here:
>
>>>> from distutils import sysconfig
>>>> sysconfig.get_makefile_filename()

This gives:

In [1]: from distutils import sysconfig

In [2]: sysconfig.get_makefile_filename()
Out[2]: '/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/profile/eoul/lib/python2.7/config/Makefile'



>
> You can edit that file, though obviously that's a bit of a hack.

It contains the lines:

CPPFLAGS=   -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
LDFLAGS=    -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu


So indeed this is causing it. I think this comes from building our own
Python, which I do with:

----------------------

#!/bin/bash

set -e

export arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
export LDFLAGS="-L/usr/lib/$arch -L/lib/$arch"
export CFLAGS="-I/usr/include/$arch"
export CPPFLAGS="-I/usr/include/$arch"
# Fix for #21:
export HAS_HG="no"
./configure --prefix=${PYTHONHPC_PREFIX}

---------------------------

And this is a bit of a hack too, Ubuntu specific etc. I think I should
start fixing things here.

It just wouldn't occur to me, that remains of how I built Python would
bite me later when building
matplotlib.

So to test if modifying the Python makefile fixes it, I did:

--- Makefile.old    2013-05-06 16:26:25.426440205 -0600
+++ Makefile    2013-05-06 16:27:05.282439550 -0600
@@ -73,8 +73,8 @@
 # Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py to
 # be able to build extension modules using the directories specified in the
 # environment variables
-CPPFLAGS=    -I. -IInclude -I$(srcdir)/Include -I/usr/include/x86_64-linux-gnu
-LDFLAGS=    -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu
+CPPFLAGS=    -I. -IInclude -I$(srcdir)/Include
+LDFLAGS=
 LDLAST=
 SGI_ABI=
 CCSHARED=    -fPIC

but mpl build system still shows the system one. The _png.so is built with:

[matplotlib] g++ -pthread -shared
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/lib
-L/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-Wl,-rpath=/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include
-I/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/freetype/lfqj/include/freetype2
build/temp.linux-x86_64-2.7/src/_png.o
build/temp.linux-x86_64-2.7/src/mplutils.o
build/temp.linux-x86_64-2.7/CXX/IndirectPythonInterface.o
build/temp.linux-x86_64-2.7/CXX/cxxsupport.o
build/temp.linux-x86_64-2.7/CXX/cxx_extensions.o
build/temp.linux-x86_64-2.7/CXX/cxxextensions.o -L/usr/local/lib
-L/usr/lib -lpng12 -lz -lstdc++ -lm -o
build/lib.linux-x86_64-2.7/matplotlib/_png.so

Which in my opinion looks good -- my own version of PNG is offered
first on the gcc command line. But the -lpng12 part spoils it --- that
forces gcc to use the systemone, because my own version is newer.


So I think that part of the problem gets fixed by modifying the Python
Makefile, but the other part of the problem is how to force distutils
to look for my PNG version before the systemwide. Any ideas?

Maybe it is something that is added by the mpl setup.py script.

>
> I've run into this problem before, and there doesn't seem to be any good way
> around it -- i.e. there doesn't seem to be a way to insert local environment
> variables in front of the global Python configuration.  Reason number #47
> why distutils is a poor build system for C/C++ code.

This is amazingly broken.

Ondrej

My understanding is that distutils builds up the commandline arguments 
for gcc in this order:

   1) From Python's Makefile.
   2) From environment variables
   3) From whatever was added by the setup.py script

It looks like you have some extra stuff in (1), namely

-L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu

You can find the Python Makefile that is being used to source this 
information here:

 >>> from distutils import sysconfig
 >>> sysconfig.get_makefile_filename()

You can edit that file, though obviously that's a bit of a hack.

I've run into this problem before, and there doesn't seem to be any good 
way around it -- i.e. there doesn't seem to be a way to insert local 
environment variables in front of the global Python configuration.  
Reason number #47 why distutils is a poor build system for C/C++ code.

Mike


On 05/06/2013 05:03 PM, Ondřej Čertík wrote:
> On Mon, May 6, 2013 at 7:18 AM, Michael Droettboom <md...@st...> wrote:
>> On 05/03/2013 02:41 PM, Ondřej Čertík wrote:
>>> Hi,
>>>
>>> As part of building matplotlib for the one python based distribution [1],
>>> I want to always link against our own version of libpng, even if there
>>> is some other systemwide version available. I am on linux (Ubuntu).
>>>
>>> Currently, here is what I am doing:
>>>
>>> CFLAGS="-I$PNG/include -I$FREETYPE/include
>>> -I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
>>> -Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
>>> $PYTHON/bin/python setup.py install
>> This *should* work. Can you provide a full build log of a clean build?
> Sure:
>
> https://gist.github.com/certik/5528134
>
> The build was produced by the "build.sh" script, also in the gist.
>
> On the line 48 (https://gist.github.com/certik/5528134#file-mpl_log-txt-L48)
> you can see where our own PNG lib is:
>
> [matplotlib] lrwxrwxrwx 1 ondrej cnls 11 May  3 11:48
> /auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng.so
> -> libpng16.so
> [matplotlib] lrwxrwxrwx 1 ondrej cnls 18 May  3 11:48
> /auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng16.so
> -> libpng16.so.16.2.0
>
> as printed by the line 5 in build.sh:
>
> echo "Our PNG library:"
> ls -l $PNG/lib/libpng*.so
>
>
> The actual mpl build starts at the line 52
> (https://gist.github.com/certik/5528134#file-mpl_log-txt-L52). As you
> can see, it found the systemwide PNG lib:
>
> [matplotlib] OPTIONAL BACKEND DEPENDENCIES
> [matplotlib]                 libpng: 1.2.46
>
> and just to verify this, at the line 2636
> (https://gist.github.com/certik/5528134#file-mpl_log-txt-L2636) I
> print:
>
> echo "The linked PNG library:"
> ldd $PYTHON/lib/python2.7/site-packages/matplotlib/_png.so
>
> which gives:
>
> [matplotlib] The linked PNG library:
> [matplotlib] linux-vdso.so.1 => (0x00007fffd8bc1000)
> [matplotlib] libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0
> (0x00007f1fd0c0a000)
> [matplotlib] libstdc++.so.6 =>
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1fd090a000)
> [matplotlib] libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x00007f1fd06f3000)
> [matplotlib] libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f1fd04d6000)
> [matplotlib] libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1fd0117000)
> [matplotlib] libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1fcfeff000)
> [matplotlib] libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1fcfc03000)
> [matplotlib] /lib64/ld-linux-x86-64.so.2 (0x00007f1fd107e000)
>
> So the systemwide png /lib/x86_64-linux-gnu/libpng12.so.0 is linked instead:
>
> ondrej@kittiwake:~$ ls -l /lib/x86_64-linux-gnu/libpng12.so.0
> lrwxrwxrwx 1 root root 18 Apr  5  2012
> /lib/x86_64-linux-gnu/libpng12.so.0 -> libpng12.so.0.46.0
>
> as you can see, it is exactly the one as advertised by the mpl build
> info above. So the mpl build seems consistent,
> and the bug is that it finds the systemwide version before our own version.
>
> Ondrej

On Mon, May 6, 2013 at 7:18 AM, Michael Droettboom <md...@st...> wrote:
> On 05/03/2013 02:41 PM, Ondřej Čertík wrote:
>> Hi,
>>
>> As part of building matplotlib for the one python based distribution [1],
>> I want to always link against our own version of libpng, even if there
>> is some other systemwide version available. I am on linux (Ubuntu).
>>
>> Currently, here is what I am doing:
>>
>> CFLAGS="-I$PNG/include -I$FREETYPE/include
>> -I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
>> -Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
>> $PYTHON/bin/python setup.py install
> This *should* work. Can you provide a full build log of a clean build?

Sure:

https://gist.github.com/certik/5528134

The build was produced by the "build.sh" script, also in the gist.

On the line 48 (https://gist.github.com/certik/5528134#file-mpl_log-txt-L48)
you can see where our own PNG lib is:

[matplotlib] lrwxrwxrwx 1 ondrej cnls 11 May  3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng.so
-> libpng16.so
[matplotlib] lrwxrwxrwx 1 ondrej cnls 18 May  3 11:48
/auto/nest/nest/u/ondrej/repos/python-hpcmp2/opt/png/qhle/lib/libpng16.so
-> libpng16.so.16.2.0

as printed by the line 5 in build.sh:

echo "Our PNG library:"
ls -l $PNG/lib/libpng*.so


The actual mpl build starts at the line 52
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L52). As you
can see, it found the systemwide PNG lib:

[matplotlib] OPTIONAL BACKEND DEPENDENCIES
[matplotlib]                 libpng: 1.2.46

and just to verify this, at the line 2636
(https://gist.github.com/certik/5528134#file-mpl_log-txt-L2636) I
print:

echo "The linked PNG library:"
ldd $PYTHON/lib/python2.7/site-packages/matplotlib/_png.so

which gives:

[matplotlib] The linked PNG library:
[matplotlib] linux-vdso.so.1 => (0x00007fffd8bc1000)
[matplotlib] libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0
(0x00007f1fd0c0a000)
[matplotlib] libstdc++.so.6 =>
/usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1fd090a000)
[matplotlib] libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f1fd06f3000)
[matplotlib] libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1fd04d6000)
[matplotlib] libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1fd0117000)
[matplotlib] libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1fcfeff000)
[matplotlib] libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1fcfc03000)
[matplotlib] /lib64/ld-linux-x86-64.so.2 (0x00007f1fd107e000)

So the systemwide png /lib/x86_64-linux-gnu/libpng12.so.0 is linked instead:

ondrej@kittiwake:~$ ls -l /lib/x86_64-linux-gnu/libpng12.so.0
lrwxrwxrwx 1 root root 18 Apr  5  2012
/lib/x86_64-linux-gnu/libpng12.so.0 -> libpng12.so.0.46.0

as you can see, it is exactly the one as advertised by the mpl build
info above. So the mpl build seems consistent,
and the bug is that it finds the systemwide version before our own version.

Ondrej

On 05/03/2013 02:41 PM, Ondřej Čertík wrote:
> Hi,
>
> As part of building matplotlib for the one python based distribution [1],
> I want to always link against our own version of libpng, even if there
> is some other systemwide version available. I am on linux (Ubuntu).
>
> Currently, here is what I am doing:
>
> CFLAGS="-I$PNG/include -I$FREETYPE/include
> -I$FREETYPE/include/freetype2" LDFLAGS="-L$FREETYPE/lib -L$PNG/lib
> -Wl,-rpath=$PNG/lib" $PYTHON/bin/python setup.py build
> $PYTHON/bin/python setup.py install
This *should* work. Can you provide a full build log of a clean build?

Mike

 Dear Python users,

I am having difficulty with numerically scaling to match line coordinates vs grid cell size coordinates. I want to calculate the following function: F =  distance_of_crossed_line x intersected_cell_value
The problem here is that when I calculate crossed_line_length in line coordinates that will unmatch vs grid coordinate, which is also another x,y with step e.g., dx=dy=2.5 each grid size. I want to do numerical calculation , say, F(distance, intersected_grid_value) function where, intersected_grid_value - values in intersected grid, distance - intersected_line_length (given below or can be found  http://stackoverflow.com/questions/16377826/distance-for-each-intersected-points-of-a-line-in-increased-order-in-2d-coordina )

import numpy as np
import scipy as sp

def distance_of_crossed_line(x0, x1, y0, y1):
       # slope
       m = (y1 - y0) / (x1 - x0)
       # Boundary of the selected points
       x_ceil = np.ceil(min(x0, x1))
       x_floor = np.floor(max(x0, x1))
       y_ceil = np.ceil(min(y0, y1))
       y_floor = np.floor(max(y0, y1))
       # calculate all intersected x coordinate
       x = np.arange(x_ceil, x_floor + 1)
       y = m * (x - x0) + y0
       ax = zip(x, y)
       # calculate all intersected y coordinate
       y = np.arange(y_ceil, y_floor + 1)
       x = (y - y0) / m + x0
       ax.extend(zip(x, y))
       ax.append((x0, y0))
       ax.append((x1, y1))
       ax.sort()

       # Transpose
       ax = np.array(ax).T
       # Calculate difference of intersections in X
       dist_x = np.diff(ax[0])
       # Calculate difference of intersections in Y
       dist_y = np.diff(ax[1])
       return np.sqrt(dist_x**2 + dist_y**2)

# PLEASE, note that line points are different from 2D array axis. they should be matched with each other. 
# 2D array.
d_array = np.array[[4.5, 4.5, 4.5, 3.4, 2.5],[ 3.9, 4.5, 5.2, 4.5, 3.4],[3.9, 3.9, 2.5, 2.2, 1.9]]

# Two sample points as a line
x = np.array([ -80, -40 ])
y = np.array([ 60, 55 ])

# The problem: 
F = intersected_line_length * array_grid_values_where_line_crossed_area
* It is not necessary for me to overlay lines onto grid cells properly, JUST, I need to calculate numerically accurate F function Thanks for the answer and guidance in advance,

-- 
Bakhtiyor Zokhidov

Hi Tony, thanks for the reply.

I was using 1.2.0 and just upgraded to 1.2.1 but the problem persists.  
I ran the example code from the link and it hangs after 350-400  
frames. Also, I got an error when running the code as it is posted and  
had to add:

import matplotlib
matplotlib.use("Agg")

to get it to work. Is this an incorrect setting I'm using?

Colin

Quoting Tony Yu <ts...@gm...>:

> On Sun, May 5, 2013 at 7:12 PM, Colin McAuliffe <cj...@co...>wrote:
>
>> Hello all,
>>
>> I'm having an issue with FuncAnimation with ffmpeg. For a small number
>> of frames everything works fine, but for some reason with more than
>> 600 frames the program hangs indefinitely. Is there some kind of frame
>> limit for FuncAmimation or is there a mistake in the arguements I am
>> passing?
>>
>
> Hi Colin,
>
> Which version of Matplotlib are you running? This sounds similar to a bug
> addressed by the following change:
>
> https://github.com/matplotlib/matplotlib/pull/989
>
> This change has already made it into the current release (v1.2).
>
> Best,
> -Tony
>



-- 
Colin McAuliffe
PhD Candidate
Columbia University
Department of Civil Engineering and Engineering Mechanics

On Sun, May 5, 2013 at 7:12 PM, Colin McAuliffe <cj...@co...>wrote:

> Hello all,
>
> I'm having an issue with FuncAnimation with ffmpeg. For a small number
> of frames everything works fine, but for some reason with more than
> 600 frames the program hangs indefinitely. Is there some kind of frame
> limit for FuncAmimation or is there a mistake in the arguements I am
> passing?
>

Hi Colin,

Which version of Matplotlib are you running? This sounds similar to a bug
addressed by the following change:

https://github.com/matplotlib/matplotlib/pull/989

This change has already made it into the current release (v1.2).

Best,
-Tony

Hello all,

I'm having an issue with FuncAnimation with ffmpeg. For a small number  
of frames everything works fine, but for some reason with more than  
600 frames the program hangs indefinitely. Is there some kind of frame  
limit for FuncAmimation or is there a mistake in the arguements I am  
passing?

ani = ani.FuncAnimation(fig, animate,np.range(1,700),  
interval=25,blit=True, init_func=init)

Thanks and all the best
Colin



-- 
Colin McAuliffe
PhD Candidate
Columbia University
Department of Civil Engineering and Engineering Mechanics