Python Github Actions CI workflow failing because numpy module can't be found

Question

I'm getting a baffling error trying to test a Python package with Github Actions. What's really puzzling is that all this worked several months ago. (I suspect this may have something to do with changes in pip post-version-22.)

The gist of the problem is that the setup.py script is failing because Python apparently can't import the "numpy" module, even though numpy is specified in the requirements.txt file, and even when it's also explicitly imported just prior to running setup.py. (The setup is "ubuntu-latest".)

The module is located here https://github.com/perwin/pyimfit

The Github Actions workflow YAML file is here (note that I've simplified it so that it only tests Python 3.11, but the same error shows up for other versions of Python): https://github.com/perwin/pyimfit/blob/master/.github/workflows/python-package.yml

name: PyImfit CI

on:
  push:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        # use quotes to avoid YAML processor interpreting version numbers as *numbers*
        # and e.g. trying to turn 3.10 into 3.1
#        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
        python-version: ['3.11']
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        # pip version 23.x breaks editable installs in this scenario (unable to import
        # previously installed modules such as numpy), for some reason
        # python -m pip install --upgrade pip==22.3.1
        python -m pip install --upgrade pip
        python -m pip install flake8 astropy pytest scons
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
        python -c "import setuptools; print(setuptools.__version__)"
    - name: Build and test import of package
      run: |
        python -m pip install numpy
        python -c "import numpy; print(numpy.__version__)"
        python -m pip install -e .
        python -c "import pyimfit"  
    - name: Test with pytest
      run: |
        cd pyimfit/tests ; pytest

The "Install dependencies" section of the workflow executes just fine.

The "Build and test import of package" section fails because Python apparently can't find the "numpy" module when running the setup.py script (as called by "pip install -e ." command).

Here's some of the output from the failed workflow, with a few added comments:

[COMMENT: this is some test code, to see if we can install and import numpy]
  python -m pip install numpy
  python -c "import numpy; print(numpy.__version__)"
[COMMENT: here is where we try to build and then test import our package]
  python -m pip install -e .
  python -c "import pyimfit"  
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.9/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.9/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.9/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.9/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.9/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.9/x64/lib
Requirement already satisfied: numpy in /opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages (1.26.4)
1.26.4
[COMMENT: so at this point it looks like python *can* find and import numpy]

[COMMENT: now we start running "pip install -e"]
Obtaining file:///home/runner/work/pyimfit/pyimfit
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-2dcc_aht/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 448, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-2dcc_aht/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-2dcc_aht/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-2dcc_aht/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-2dcc_aht/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 25, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

I've varied which versions of Python were tested (3.7-3.12); I've also tried forcing an update to pip 24.0, leaving it the default Github Actions version, and downgrading to pip 22.3.1 (which was working in February, the last time the workflow was run successfully).

What I was expecting see was something like this (from the aforementioned last successful CI test):

  python -m pip install -e .
  python -c "import pyimfit"  
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.10.14/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib
Obtaining file:///home/runner/work/pyimfit/pyimfit
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: numpy in /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages (from pyimfit==0.13.0) (1.26.4)
Requirement already satisfied: scipy in /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages (from pyimfit==0.13.0) (1.13.0)
Installing collected packages: pyimfit
  Running setup.py develop for pyimfit
Successfully installed pyimfit-0.13.0

Yasien Dwieb · Accepted Answer · 2024-04-26 09:01:05Z

0

This seems a known issue For numpy with setuptools. The recommended fix is to add numpy to the setup_requires in your setup.py file.

More related resources about this exact issue:

Similar issue on Stackoverflow: Installing numpy as a dependency with setuptools
The corresponding issue in numpy repo: https://github.com/numpy/numpy/issues/2434

answered Apr 26, 2024 at 9:01

Yasien Dwieb

1692 silver badges3 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Peter Erwin Over a year ago

Unfortunately, when I try that (adding numpy to setup_requires in the setup.py file, I get exactly the same error.

Collectives™ on Stack Overflow

Python Github Actions CI workflow failing because numpy module can't be found

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related