Embedded python in C : Is there a way to import numpy properly from zipped python archive?

Question

Seems like I can not import numpy properly from within C application which embeds python, when using local .zip python file containing all the python files and binaries rather then standard python installation. This is the error I'm getting :

zipimport: zlib available
python27.zip\Lib\site-packages\numpy_globals.pyc has bad mtime
zipimport: zlib available
import numpy._globals # loaded from Zip python27.zip\Lib\site-packages\numpy_globals.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy__config__.pyc has bad mtime
zipimport: zlib available
import numpy.config # loaded from Zip python27.zip\Lib\site-packages\numpy__config__.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\version.pyc has bad mtime
zipimport: zlib available
import numpy.version # loaded from Zip python27.zip\Lib\site-packages\numpy\version.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy_import_tools.pyc has bad mtime
zipimport: zlib available import numpy._import_tools # loaded from Zip python27.zip\Lib\site-packages\numpy_import_tools.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\add_newdocs.pyc has bad mtime
zipimport: zlib available
zipimport: zlib available
import math # builtin
zipimport: zlib available
import numpy.lib.info # loaded from Zip python27.zip\Lib\site-packages\numpy\lib\info.pyc
zipimport: zlib available
zipimport: zlib available
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\core\info.pyc has bad mtime
zipimport: zlib available
import numpy.core.info # loaded from Zip python27.zip\Lib\site-packages\numpy\core\info.py
import numpy.core # loaded from Zip python27.zip\Lib\site-packages\numpy\core__init__.pyc
import numpy.lib.type_check # loaded from Zip python27.zip\Lib\site-packages\numpy\lib\type_check.pyc
import numpy.lib # loaded from Zip python27.zip\Lib\site-packages\numpy\lib__init__.pyc
import numpy.add_newdocs # loaded from Zip python27.zip\Lib\site-packages\numpy\add_newdocs.py
import numpy # loaded from Zip python27.zip\Lib\site-packages\numpy__init__.py
cannot import name multiarray

But when I extract abovementioned .zip file, and run python.exe interpreter and import numpy, everything works fine.

I've build python 2.7.13 from source, x86 Release. After that installed numpy-1.11.3+mkl-cp27-cp27m-win32.whl file from here, then made a python .zip archive with all neccessary files following common folder structure.

This is how my C code looks like :

int main(int argc, char **argv)
{
    Py_VerboseFlag++;
    Py_NoSiteFlag++;

    Py_SetProgramName(argv[0]);
    Py_SetPythonHome(".");

    Py_Initialize();

    PyRun_SimpleString("import sys");
    PyRun_SimpleString("sys.path = ['.','python27.zip','python27.zip/DLLs','python27.zip/Lib', 'python27.zip/Lib/site-packages']");;
    PyRun_SimpleString("print sys.path");

    printf("\n");

    char filename[_MAX_PATH];
    _fullpath(filename, "mod1.py", _MAX_PATH);

    PyObject* main_module = PyImport_AddModule("__main__");
    PyObject* main_dict = PyModule_GetDict(main_module);
    PyObject *pyFileObj = PyFile_FromString(filename, "r");

    if (pyFileObj == NULL) {
        return -1;
    }

    FILE *pFile = PyFile_AsFile(pyFileObj);

    if (pFile == NULL)
        return -1;

    PyObject *result = PyRun_File(pFile, filename, Py_file_input, main_dict, main_dict);

    if (!result)
        print_error();

    printf("\n\n");


    Py_Finalize();
    getchar();

    return 0;
}

module mod1.py, I'm executing contains this :

import time
import json
import numpy

if __name__ == "__main__":
    print 'Success'

I'm aware that that error comes from the fact that python can not load multiarray.pyd which is shared library.Do I need to handle this scenario separately when it comes to import?
Is it possible to import numpy properly from zipped, non-standard python installation from C application?

EDIT : Forgot to mention following :
- My OS is Windows 10 x64 Version 1511 (OS Build 10586.545)
- When I extract .zip archive in my Release folder where my C app binary resides, everything works fine, numpy is loaded properly with following sys.path

PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path = ['.', 'DLLs', 'Lib', 'Lib\\site-packages', 'Sources']");
PyRun_SimpleString("print sys.path");

EDIT 2: Also, thought to check exact version of python27.dll the multiarray.pyd is linked to (according to other posts about similar problems) and it turned out it is different then mine - it's 2.7.8. No problem, extracted specific python27.dll v2.7.8 binary from the .msi package and replaced mine which was 2.7.13, but still no luck. It is definitely something about zipimport module.

You cannot load a shared library from a zip file. This is a limitation of the operating system and applies to Windows, Linux and Mac OS. It is not an issue with Python or its zip file importer. Since Numpy contains C extensions you cannot bundle it in a zip executable. — JacaByte
– JacaByte, Commented Feb 18, 2018 at 23:23
Thanks for the answer, I've already learned that the hard way.However, I think I made it (on Win) by utilizing MemoryModule lib written by Joachim Bauch by simply decompressing archive into memory and load stuff from there.Sample code here : github.com/Civa/continuum/blob/master/src/continuum/runtime/… (WARNING : poorly written library with messy code since I'm still learning C) — civa
– civa, Commented Feb 19, 2018 at 19:56
Interesting, I ended up doing something similar for Linux; conceptually it's a wrapper that copies the binary into a folder in /tmp, adds the folder to sys.path and evals from mymodule import * to load the extension. I didn't know about this loader. — JacaByte
– JacaByte, Commented Feb 20, 2018 at 3:33
ah sorry @JasonArg123, I've migrated all the stuff from github and might have deleted obsolete repos in the process. As soon as I find the source I will upload it to the gitlab — civa
– civa, Commented Jun 4, 2020 at 10:00
@JasonArg123 here is the old repo I found. If you want to see how loader works, just search for loader.c. Note that, this code is VERY messy. Since this is written for windows I can't be of much help since I swithced to LInux long time ago, sorry. Please feel free to do anything you want with the code as there is no license attached. Hope you will find it useful. Cheers, civa — civa
– civa, Commented Feb 17, 2021 at 12:24

Alex W · Accepted Answer · 2021-03-31 16:41:44Z

1

The best way is probably to unzip it for now. The zipimport module is what Python uses to load modules from zip files and unfortunately it disallows the import of dynamic compiled code (probably due to security concerns, consistent with PEP 273):

Any files may be present in the zip archive, but only files *.py and .py[co] are available for import. Zip import of dynamic modules (.pyd, *.so) is disallowed.

Because the wheels provided will be platform wheels, you will also need to remove the platform extension from the .so filenames which are added by Cython as part of adherence to PEP 3149. I.e., foo.cpython-XYm.so would need to be renamed to foo.so. (Since your question asks about Python 2, this does not apply but it does going forward for Python 3)

Lastly, the unzipped folders will need to be added to the PYTHONPATH if they are not already on it.

edited Mar 31, 2021 at 16:41

answered Mar 31, 2021 at 14:22

Alex W

38.5k13 gold badges114 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

civa Over a year ago

thanks for the reply, but I abandoned that project long time ago.

Alex W Over a year ago

No problem, had this issue on Spark cluster so I did an in-depth investigation and wanted others to not have to

Collectives™ on Stack Overflow

Embedded python in C : Is there a way to import numpy properly from zipped python archive?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related