setup.py not installing swig extension module

Question

I'm struggling to figure out how to copy the wrapper generated by swig at the same level than the swig shared library. Consider this tree structure:

│   .gitignore
│   setup.py
│
├───hello
├───src
│       hello.c
│       hello.h
│       hello.i
│
└───test
        test_hello.py

and this setup.py:

import os
import sys

from setuptools import setup, find_packages, Extension
from setuptools.command.build_py import build_py as _build_py


class build_py(_build_py):
    def run(self):
        self.run_command("build_ext")
        return super().run()


setup(
    name='hello_world',
    version='0.1',
    cmdclass={'build_py': build_py},
    packages=["hello"],
    ext_modules=[
        Extension(
            'hello._hello',
            [
                'src/hello.i',
                'src/hello.c'
            ],
            include_dirs=[
                "src",
            ],
            depends=[
                'src/hello.h'
            ],
        )
    ],
    py_modules=[
        "hello"
    ],
)

When I do pip install . I'll get this content on site-packages:

>tree /f d:\virtual_envs\py364_32\Lib\site-packages\hello                            
D:\VIRTUAL_ENVS\PY364_32\LIB\SITE-PACKAGES\HELLO                                                                 
    _hello.cp36-win32.pyd                                                                                        


>tree /f d:\virtual_envs\py364_32\Lib\site-packages\hello_world-0.1.dist-info        
D:\VIRTUAL_ENVS\PY364_32\LIB\SITE-PACKAGES\HELLO_WORLD-0.1.DIST-INFO                                             
    INSTALLER                                                                                                    
    METADATA                                                                                                     
    RECORD                                                                                                       
    top_level.txt                                                                                                
    WHEEL

As you can see hello.py (the file generated by swig) hasn't been copied in site-packages.

Thing is, I've already tried many answers from the below similars posts:

Unfortunately, the question still remains unsolved.

QUESTION: How can I fix my current setup.py so the swig wrapper will be copied at the same level than the .pyd file?

What is hello on project root level? Is it another python package to be included? If everything you want to include is under src, then map hello sources to src via package_dir={'hello': 'src'} and remove py_modules. — hoefling
– hoefling, Commented Jun 12, 2019 at 21:27
@hoefling hello on project root level it's a python package that would probably include the generated wrapper as well as python helpers on top of it using the wrapper — BPL
– BPL, Commented Jun 13, 2019 at 7:06
I'd rename it to something else to avoid the name clash as AFAIK distutils cannot map multiple source locations to a single package. Aside from that, the suggestion in the comment above should work. — hoefling
– hoefling, Commented Jun 13, 2019 at 11:44

fpbhb · Accepted Answer · 2019-06-20 21:21:22Z

1

setuptools cannot do this the way you want: it will look for py_modules only where setup.py is located. The easiest way IMHO is keeping the SWIG modules where you want them in the namespace/directory structure: rename src to hello, and add hello/__init__.py (may be empty or simply include everything from hello.hello), leaving you with this tree:

$ tree .
.
├── hello
│   ├── __init__.py
│   ├── _hello.cpython-37m-darwin.so
│   ├── hello.c
│   ├── hello.h
│   ├── hello.i
│   ├── hello.py
│   └── hello_wrap.c
└── setup.py

Remove py_modules from setup.py. The "hello" in the package list will make setuptools pick up the whole package, and include __init__.py and the generated hello.py:

import os
import sys

from setuptools import setup, find_packages, Extension
from setuptools.command.build_py import build_py as _build_py


class build_py(_build_py):
    def run(self):
        self.run_command("build_ext")
        return super().run()


setup(
    name='hello_world',
    version='0.1',
    cmdclass={'build_py': build_py},
    packages = ["hello"],
    ext_modules=[
        Extension(
            'hello._hello',
            [
                'hello/hello.i',
                'hello/hello.c'
            ],
            include_dirs=[
                "hello",
            ],
            depends=[
                'hello/hello.h'
            ],
        )
    ],
)

This way, also .egg-linking the package works (python setup.py develop), so you can link the package under development into a venv or so. This is also the reason for the way setuptools (and distutils) works: the dev sandbox should be structured in a way that allows running the code directly from it, without moving modules around.

The SWIG-generated hello.py and the generated extension _hello will then live under hello:

>>> from hello import hello, _hello
>>> print(hello)
<module 'hello.hello' from '~/so56562132/hello/hello.py'>
>>> print(_hello)
<module 'hello._hello' from '~/so56562132/hello/_hello.cpython-37m-darwin.so'>

(as you can see from the extension filename, I am on a Mac right now, but this works exactly the same under Windows)

Also, beyond packaging, there's more useful information about SWIG and Python namespaces and packages in the SWIG manual: http://swig.org/Doc4.0/Python.html#Python_nn72

answered Jun 20, 2019 at 21:21

fpbhb

1,51912 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

BPL Over a year ago

Thanks for the answer, tomorrow I'll check this method with some real case but I must to say I really dislike the fact of mixing both source & build (generated) stuff in the same folder. Do you know why I've actually put my original dummy example as a source folder? Because the goal was to come up with a standard practice that'd allow me to wrap up c++ packages cloned as submodules in my wrapper repo. In fact, you can see in my example i've mixed swig files with source files, probably they should also live in different folders... Or maybe if using properly .gitignore isn't that bad, mmm :/

fpbhb Over a year ago

Fair point. For the software I work on, we keep sources separate and (lots of!) 3rd-party dependencies as well, and have a build system separate from setuptools to build the extensions; it then becomes a two-step process, though. Building, packaging etc. is driven by some top-level automation. Setuptools/distutils simply is not made for more complex setups.

BPL Over a year ago

Yeah, makes total sense... in fact, I also got custom build tools myself to build swig stuff... problem with these home-made tools is they're just available for yourself. That's why I wanted to figure out what the "best standard" practices to deliver c extension modules using swig :) . Unfortunately, when you look at swig packages available at github there isn't any "common" way about it... that's the reason of why I've opened this thread in the first place ;)

Collectives™ on Stack Overflow

setup.py not installing swig extension module

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related