Packaging Python module with source C/C++ files and compile on install

I have a C library which I have ported to Python3. The Python module loads C code from the shared library.

However, instead of building shared library for for each platform, I plan to package the C/C++ code which can be compiled at the user end when installing the module.

I am using setuptools and I tried using ext_modules as below. However, instead of packaging c files, it locally build it and adding it to the final package (.whl) file which defeats the purpose.

ext_modules=[
        Extension(
            # the qualified name of the extension module to build
            'mymodule',
            # the files to compile into our module relative to ``setup.py``
            ['library.cpp'],
        ),
    ],

I also tried using cmdclass to run post installation build but it only works on local installation and not when installing from PyPi. A discussion here

Pip not picking up a custom install cmdclass

https://chat.stackoverflow.com/rooms/150536/discussion-between-swiftsnamesake-and-collins-a

Thanks