13

I am unable to import numpy 1.19.1 in AWS Lambda with python3.8 on AWS Lambda

I am using the following dependencies:

  • pandas 1.1.0
  • pyarrow 1.0.0
  • numpy 1.19.1
  • psycopg2 2.8.5

Because I work on a windows environment, I created an EC2 Linux instance installed python3.8 and downloaded all required libraries, then I added them into the project, but the moment I try to import pandas I get the following:

[ERROR] ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "/var/lang/bin/python3.8",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.2" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "/var/task/src/py38-lib-test.py", line 28, in py38test
    import pandas
  File "/tmp/lib/pandas/__init__.py", line 16, in <module>
    raise ImportError(END RequestId: 07762380-1fc4)

Lastly, I noticed AWS Lambda provides a layer with numpy and sci-kit, I tried removing my numpy version but kept the rest and added the layer to the function, but the same error occurs.

Thanks in advance your comments.

Improve this question
3
  • If you just use AWS's provided layer, without any of yours, does it work? Commented Aug 13, 2020 at 22:17
  • The lib is probably not compatible with the current lambda runtime Commented Aug 14, 2020 at 0:53
  • @Marcin, it works alone, but when using pandas and pyarrow, it does not work. I needed them to read a .parquet file Commented Aug 14, 2020 at 13:32

6 Answers 6

Reset to default
5

I use the layer provided by Klayers to solve the problem.

Suppose you're running python 3.8 in us-east-1 region, according to this Klayers document, you can use arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p38-numpy:9 as your layer so that you can run import numpy in the lambda function.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

this work for me , but with set in sls py38 and numpy=1.21.2 it didn't work either
Please note that currently we have no Pandas layer provided by Klayers compatible with Python 3.9. If you need Pandas, use Python 3.8 runtime.
3

AWS Lambda function don't work this way. If you open the Pandas package it'll be having the Numpy package with them but they would not work. The easy solution is to first download the required packages separately depending upon your python version and work enviroment from this site, unzip them and add them to your project directory. Create a .zip of your project and deploy it on AWS Lambda function. It'll work this way. You can refer to this site in order to follow the complete procedure.

Improve this answer

Comments

2

Is your ec2 instance an amazon linux2 machine? You could also try building and run a docker image for amazon linux 2 and get the python libs compatible to the environment you need in your Lambda, by volume mounting to your host.

Something similat to docker lambda:

https://github.com/lambci/docker-lambda/tree/master/python3.8

Improve this answer

2 Comments

I think this is the issue (linux version), it's complicated for me as the environment is very restrictive. I cannot use docker or even get software properly. Someone in the team got the right libraries somehow and we were able to use them. thanks for the help
This answer should deserve more votes.
1

I had the same issue; I tried packaging all my libraries with my base code; I tried making a custom lambda layer by separating numpy and pandas libs. Nothing worked.

In the end, adding a default AWS Layer to my lambda function that comes with numpy already installed solved the issue.

  1. Remove numpy and pandas libraries from the your lambda package and zip the contents
  2. In your lambda function, click "Add a layer" and select the AWSSDKPandas option from the AWS default layers.
  3. Upload your modified lambda package and run the code.
Improve this answer

2 Comments

Welcome to SO! Could you please more information in the answer e.g. link to the documentation for AWS SDK Pandas?
Would you mind sharing an example?
0

Could you check the Python version you are using while packaging the libraries and Python version you are trying to run.

I got the issue when I packaged libraries with python 3.11 and lambda was on python 3.9. I packaged libraries with python 3.9 and the issue is resolved.

Improve this answer

Comments

0

Use Amazon Linux

To install Python packages and build Lambda deployment package or layers, use Amazon Linux.

Python packages such as numpy has native C implementation included. Hence, as described in No module named 'numpy.core._multiarray_umath' #21678, we cannot simply copy and the python packages installed in an OS/distribution and copy them to different Amazon Linux Lambda environment.

You cannot copy around the numpy/scipy directories. There are support libraries installed with the package that will not be found if you simply copy. You should use python -m pip instead of pip3 since you may have more than one python installed on your machine. When you type pip3 install numpy scipy, it is not clear into which python environment you will install numpy and scipy. When you specify python -m pip, then there is less room for problems.
Try opening a command prompt and reinstalling numpy and scipy:

python -m pip install --force-reinstall numpy scipy

As instructed in AWS Lambda Layers with Python Package, use Amazon Linux to package the python deployment modules.

As the error states the issue with how you NumPy was installed. Most probably you compiled it on your dev machines which might be windows, mac or some flavor of linux. While lambda runs on Amazon Linux. Since NumPy is a C extension it is better that you build the Python package for your panda and NumPy using Amazon Linux.

However, the fundamental issue is the poor AWS documentations that do not clearly explain such prerequisites.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.