194

I'm trying to speed up the answer here using Cython. I try to compile the code (after doing the cygwinccompiler.py hack explained here), but get a fatal error: numpy/arrayobject.h: No such file or directory...compilation terminated error. Can anyone tell me if it's a problem with my code, or some esoteric subtlety with Cython?

Below is my code. 

import numpy as np
import scipy as sp
cimport numpy as np
cimport cython

cdef inline np.ndarray[np.int, ndim=1] fbincount(np.ndarray[np.int_t, ndim=1] x):
    cdef int m = np.amax(x)+1
    cdef int n = x.size
    cdef unsigned int i
    cdef np.ndarray[np.int_t, ndim=1] c = np.zeros(m, dtype=np.int)

    for i in xrange(n):
        c[<unsigned int>x[i]] += 1

    return c

cdef packed struct Point:
    np.float64_t f0, f1

@cython.boundscheck(False)
def sparsemaker(np.ndarray[np.float_t, ndim=2] X not None,
                np.ndarray[np.float_t, ndim=2] Y not None,
                np.ndarray[np.float_t, ndim=2] Z not None):

    cdef np.ndarray[np.float64_t, ndim=1] counts, factor
    cdef np.ndarray[np.int_t, ndim=1] row, col, repeats
    cdef np.ndarray[Point] indices

    cdef int x_, y_

    _, row = np.unique(X, return_inverse=True); x_ = _.size
    _, col = np.unique(Y, return_inverse=True); y_ = _.size
    indices = np.rec.fromarrays([row,col])
    _, repeats = np.unique(indices, return_inverse=True)
    counts = 1. / fbincount(repeats)
    Z.flat *= counts.take(repeats)

    return sp.sparse.csr_matrix((Z.flat,(row,col)), shape=(x_, y_)).toarray()
Improve this question
5
  • can you add a tag for what OS you are using? Commented Feb 2, 2013 at 2:37
  • @tcaswell 64-bit Windows 7. Commented Feb 2, 2013 at 2:43
  • added the windows tag, hopefully that will help this problem be seen by people who know how to use windows (unlike me). Commented Feb 2, 2013 at 2:45
  • 1
    I found this. Some of the terminology is above my head, but i'm gonna check it out. Commented Feb 2, 2013 at 4:31
  • Does this answer your question? Make distutils look for numpy header files in the correct place Commented Apr 16, 2020 at 19:33

10 Answers 10

Reset to default
272

In your setup.py, the Extension should have the argument include_dirs=[numpy.get_include()].

Also, you are missing np.import_array() in your code.

--

Example setup.py:

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy

setup(
    ext_modules=[
        Extension("my_module", ["my_module.c"],
                  include_dirs=[numpy.get_include()]),
    ],
)

# Or, if you use cythonize() to make the ext_modules list,
# include_dirs can be passed to setup()

setup(
    ext_modules=cythonize("my_module.pyx"),
    include_dirs=[numpy.get_include()]
)
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Why would i need np.import_array()? Isn't that for the Numpy C-API?
I tried your idea, but now i get this crazy error that is too long to post here, but it starts with warning: untitled.pyx:8:49: Buffer unpacking not optimized away.
Ah, true, you probably don't need np.import_array(). I rarely write numpy-using Cython extensions without it, though, so I use it as a matter of habit. As for your other problem, what you quoted is just a warning, not an error. If you have other errors that need fixing, please make a new post.
definitely helped me building a point cloud library (PCL) python wrapper that is using Numpy and Cython, however, there are warnings Using deprecated NumPy API - is this deprecated?
include_dirs passed to setup() gets ignored in the latest distutils, it has to be passed to each Extension, at least on mac
|
54

For a one-file project like yours, another alternative is to use pyximport. You don't need to create a setup.py ... you don't need to even open a command line if you use IPython ... it's all very convenient. In your case, try running these commands in IPython or in a normal Python script:

import numpy
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"],
                              "include_dirs":numpy.get_include()},
                  reload_support=True)

import my_pyx_module

print my_pyx_module.some_function(...)
...

You may need to edit the compiler of course. This makes import and reload work the same for .pyx files as they work for .py files.

Source: http://wiki.cython.org/InstallingOnWindows

Improve this answer

4 Comments

Thanks, @SteveB. But can you elaborate a bit on what you mean by "For a one-file project like yours..."? The module above is one (albeit important) part of a larger application. How does pyximport affect my code's speed? And finally, the section here: "Since Cython 0.11, the pyximport module also has experimental compilation support for normal Python modules..." implies that it still has some kinks to work out. Can you explain that as well?
Re "experimental compilation support for normal Python modules" -- with the code I suggested above, .py modules are compiled normally (not with cython) while .pyx modules are compiled with cython. If you pass pyimport = True into pyximport.install(), then it will use cython for everything, even for example import random or import os. I don't suggest using that feature, simply because there's no compelling reason to use it, and it could create problems. It's probably used mainly by cython developers.
If pyximport works at all, it will create the exact same C code as any other method. So try it and see. I was referring to the fact that when the compilation process is sufficiently complicated, e.g. links to external system libraries, you might find that pyximport fails and you need a setup.py and cythonize to specify exactly how to build it. But the fact that your .pyx module has imports or cimports does not mean that it can't be compiled with pyximport; it may well be totally fine.
This works great! But I recommend --compiler=msvc if you already have msvc installed. Then you don't need to install mingw.
23

The error means that a numpy header file isn't being found during compilation.

Try doing export CFLAGS=-I/usr/lib/python2.7/site-packages/numpy/core/include/, and then compiling. This is a problem with a few different packages. There's a bug filed in ArchLinux for the same issue: https://bugs.archlinux.org/task/22326

Improve this answer

11 Comments

Where do i add the export line? In my setup.py file?
No, it's a shell command. Run it in your shell, then start compiling.
@NoobSaibot in the shell (where you run python setup.py) run the export .. command first. It sets the environmental variables of the shell, not anything directly to do with [pc]ython.
@tcaswell: I figured as much. I'm using cmd, and got this 'export' is not recognized as an internal or external command, operable program or batch file. error...just can't win with this one...
@NoobSaibot you are getting lunix answers for what smells like a windows problem....
|
6

As per this answer, if you have installed numpy with pip on Linux, you will need to manually set a symbolic link to /usr/include/numpy

In my case the path is:

sudo ln -s /usr/local/lib/python3.8/dist-packages/numpy/core/include/numpy/ /usr/include/numpy
Improve this answer

Comments

4

Simple

$ python
import numpy as np
np.get_include()

'/Users/mudasirhabib/Documents/projects/verticalsols/tts/venv/lib/python3.9/site-packages/numpy/core/include'*

Now in terminal enter

 export CFLAGS="-I/Users/mudasirhabib/Documents/projects/verticalsols/tts/venv/lib/python3.9/site-packages/numpy/core/include $CFLAGS"

Thats all

Improve this answer

Comments

3

If you are too lazy to write setup files and figure out the path for include directories, try cyper. It can compile your Cython code and set include_dirs for Numpy automatically.

Load your code into a string, then simply run cymodule = cyper.inline(code_string), then your function is available as cymodule.sparsemaker instantaneously. Something like this

code = open(your_pyx_file).read()
cymodule = cyper.inline(code)

cymodule.sparsemaker(...)
# do what you want with your function

You can install cyper via pip install cyper.

Improve this answer

Comments

1

Simple answer

A way simpler way is to add the path to your file distutils.cfg. It's path behalf of Windows 7 is by default C:\Python27\Lib\distutils\. You just assert the following contents and it should work out:

[build_ext]
include_dirs= C:\Python27\Lib\site-packages\numpy\core\include

Entire config file

To give you an example how the config file could look like, my entire file reads:

[build]
compiler = mingw32

[build_ext]
include_dirs= C:\Python27\Lib\site-packages\numpy\core\include
compiler = mingw32
Improve this answer

Comments

1

It should be able to do it within cythonize() function as mentioned here, but it doesn't work beacuse there is a known issue

Improve this answer

Comments

0

If you're using cythonize and see this error:

[tmp]$ echo "cimport numpy" >> a.pyx
[tmp]$ cythonize -i a.pyx
Compiling /tmp/a.pyx because it changed.
[1/1] Cythonizing /tmp/a.pyx
/tmp/a.c:1240:10: fatal error: numpy/arrayobject.h: No such file or directory
 1240 | #include "numpy/arrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/lib/ccache/gcc' failed with exit code 1

the export CFLAGS solution works for me. You can also shorten it by doing it in one step as follows (or CXXFLAGS) (assume you're using Bash):

[tmp]$ CFLAGS="-O3 -I$(python -c "import numpy; print(numpy.get_include())")" cythonize -i a.pyx
.../npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~

The -O3 is optional.

Improve this answer

Comments

-1

I hadn't sudo privileges on the server I was running and export CFLAGS didn't work with me. For sake of simplicity, I've installed Anaconda ( https://docs.anaconda.com/anaconda/install/) which creates links to all its installed packages, including Numpy. You can also install miniconda and work with environments to avoid using too much space.

Improve this answer

1 Comment

Are you sure this answers the question? Please read How to Answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.