why does python import causes an error with PyImport_ImportModule but not with python interpreter

Question

I have a python module mymodule that contains an import psutil statement. I can run it using python interpreter without any errors.
However, I fail to load it with PyImport_ImportModule("mymodule") (from python C API), due to an undefined symbol in import psutil.
I don't understand what causes this, and since the C code is part of a shared object that is aimed to load various python modules, I will appreciate any help.

I'm suspecting that this is a bigger issue than this specific module, but here are some specific details-

traceback

Traceback (most recent call last):
  File "/path/to/mymodule.py", line 7, in <module>
    import psutil
  File "/usr/local/lib/python3.8/dist-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: /usr/local/lib/python3.8/dist-packages/psutil/_psutil_linux.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyExc_RuntimeError

ldd

linux-vdso.so.1 (0x00007ffe91b5e000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f5a1b41a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5a1b3f7000)
        libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f5a1b121000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5a1af3f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5a1af24000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5a1ad30000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f5a1ad02000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5a1ace6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5a1ace0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5a1acdb000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5a1ab8c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5a1b9a8000)

psutil on python interpreter:

>>> import psutil
>>> psutil.__file__
'/usr/local/lib/python3.8/dist-packages/psutil/__init__.py'

@yoonghm thanks but this has nothing to do with import __future__ or sys.path — jsofri
– jsofri, Commented Nov 11, 2021 at 8:42
A function call to PySys_SetPath() completely overwrites the Python module path. From the standard documentation, This is a simplified interface to PyImport_ImportModuleEx() below, leaving the globals and locals arguments set to NULL and level set to 0. Perhaps you show more of your code — yoonghm
– yoonghm, Commented Nov 11, 2021 at 8:46

jsofri · Accepted Answer · 2021-11-11 09:23:55Z

2

OK, so the issue here was the way dlopen link the libraries.
For my case, updating dlopen from
dlopen(so_lib_name, RTLD_NOW)
to
dlopen(so_lib_name, RTLD_NOW|RTLD_GLOBAL)
worked!

answered Nov 11, 2021 at 9:23

jsofri

2451 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

RTC222 Over a year ago

I just had the same problem and your answer solved it for me. Thanks.

Collectives™ on Stack Overflow

why does python import causes an error with PyImport_ImportModule but not with python interpreter

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related