Numpy operations appear slow

Question

I am trying to see how fast I can invert a large matrix through numpy and am facing a rather weird conundrum. The test code is very simple:

file invert.py

from numpy.linalg import inv
import time

def invert():
 a=np.random.rand(10000,10000)
 s = time.time()
 b=inv(a)
 print time.time()-s

I am using a Windows machine so that I can run this test either through the Windows command prompt or through something like cygwin. The commands I use are exactly the same in either case:

python
import invert
invert.invert()

However when I issue the command from the win command prompt I get an elapsed time of ~30 seconds. When I do it through cygwin I get an elapsed time of ~1700 seconds! I am also trying to run the same test on a cluster node (linux) and get very slow results (~1600 seconds). I am quite baffled with this. Issuing the same python commands from the command prompt appears to be working massively faster than issuing them through cygwin. This makes no sense to me. Can someone shed some light on why this may be happening?

Thanks,

Can you reproduce this over multiple runs? The work needed for inversion might vary depending on the (random) values of the matrix. — user395760
– user395760, Commented Mar 7, 2014 at 23:06
Yes I can reproduce it over different runs and it is very consistent. Moreover, I have tried inverting matrices of different sizes and the results are always consistent with win cmd being considerably faster. — user2874390
– user2874390, Commented Mar 7, 2014 at 23:31
Are both machines using optimized blas routines? It sounds like this is not the case. You can use numpy.show_config() to see what is being used. You can also try scipy.linalg.inv() though I do not expect there to be much time difference. — Craig J Copi
– Craig J Copi, Commented Mar 7, 2014 at 23:40
I get the following about blas and lapack when I give that command on the windows machine: blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] — user2874390
– user2874390, Commented Mar 8, 2014 at 5:24
Let me add another little point. In addition to command line arguments given through win cmd and cygwin I have also tested the same program through the Eclipse IDE. When I do that I get speeds as fast as win cmd. As I said I have tested this over and over again and the issue is very consistent. I can't figure out what is different about issuing python commands through win cmd and Eclipse IDE that is making such a huge difference in the time of computation of this simple code. — user2874390
– user2874390, Commented Mar 8, 2014 at 5:32

user2874390 · Accepted Answer · 2014-03-13 00:19:04Z

1

Thanks for the answers. Although I haven't completely figured out why this is happening Craig's answer has been helpful. When I give the numpy.show_config() command from win cmd I get info about the blas and lapack libraries. However when I give the same command from cygwin I don't get that info. I can only assume that running python through cygwin is somehow not aware of where the libraries are.

answered Mar 13, 2014 at 0:19

user2874390

1092 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Numpy operations appear slow

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related