Installing numpy on Mac to work on AWS Lambda

Question

Is there a way to install numpy on a Mac so that it will work when uploaded to AWS Lambda? I have tried a variety of different ways, including using different pip versions, using easy_install, and following this post, but none of them seem to work. I also tried cloning the git repo and building from there, but I also wasn't able to get that to work (though I'm not sure if I copied the right files up after doing that)

The error I'm getting is:

Unable to import module 'lambda_function': Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, try git clean -xdf (removes all files not under version control). Otherwise reinstall numpy.

Inspired by this post, I was able to pip install numpy in a Linux environment and get it to work on Lambda.

So my question is: Is it possible to install numpy on a Mac so that it works on AWS Lambda?

Environment: MacBook Pro, MacOS 10.12.2, default python version 2.7.10

I've been testing it with a minor variation on the hello-world-python example on Lambda:

from __future__ import print_function
import numpy

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])

(Update) Extending the question: Why do some packages work and others don't?

You have to use a version built for Amazon Linux. You can't package Mac binaries in your Lambda deployment. — Mark B
– Mark B, Commented Mar 1, 2017 at 14:54
Thanks @MarkB. Do you know why that's the case for numpy and not for other packages? numpy is the only one where I've been running into this issue (so far). Asked another way - is there a way to know ahead of time which packages will require a Linux build? — Tchotchke
– Tchotchke, Commented Mar 1, 2017 at 14:55
You can unpackage the numpy whl file from the python project download files, there's a fuller answer here stackoverflow.com/questions/43877692/… — chim
– chim, Commented Oct 5, 2018 at 12:28
@Tchotchke Probably because numpy is the only package you've used which has some parts written in C. The others are probably pure python, which is generally platform independent. Whereas C needs to be compiled into a binary which is specific to a platform. — falsePockets
– falsePockets, Commented Dec 27, 2019 at 5:50

NealWalters · Accepted Answer · 2020-10-26 22:32:07Z

7

Update: the preferred approach now is to just use the AWS-provided Lambda Layer for NumPy/SciPy, which is super easy to do.

In the console, select your function and then, under the "Design" section click on "Layers". Then click "Add a Layer", and select "AWSLambda-Python37-SciPy1x" under AWS Provided (or whatever the equivalent is for the version of Python you're using).

Then you can seamlessly import numpy, scipy, etc. into your code with no issues.

10/26/2020 - Added example screenshot:

edited Oct 26, 2020 at 22:32

NealWalters

18.5k48 gold badges165 silver badges283 bronze badges

answered Sep 27, 2019 at 14:51

James Shapiro

6,3985 gold badges39 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

NealWalters Over a year ago

Added screen shots to your answer, and also people can see this link: towardsdatascience.com/…

James Shapiro · Accepted Answer · 2019-11-27 03:23:19Z

Update: Please ignore the below and see my response above (here: Installing numpy on Mac to work on AWS Lambda) about using Lambda Layers. That is a MUCH easier approach.

===========================================================================

I had this same exact issue and resolved it easily by following the advice here:

https://serverless.com/blog/serverless-python-packaging/

Note that I also had to add the lines:

package:
  exclude:
    - venv/**

... to the end of my serverless.yml to get my zip file under the size limit.

I will re-copy the instructions from the blog post below, just in case the link ever goes stale.

==============================================

Create a Lambda Function that uses numpy

install node/npm
npm install -g serverless
configure aws for your computer (install the cli, run "aws configure" and add your credentials generated from IAM). 3a. install Docker for Mac

Run the following:

serverless create \
  --template aws-python3 \
  --name numpy-test \
  --path numpy-test

cd numpy-test
virtualenv venv --python=python3
source venv/bin/activate

edit handler.py and replace its contents with:

handler.py

import numpy as np
def main(event, context):
    a = np.arange(15).reshape(3, 5)
    print("Your numpy array:")
    print(a)


if __name__ == "__main__":
    main('', '')

6.

Run the following:

pip3 install numpy
pip3 freeze > requirements.txt
cat requirements.txt

replace serverless.yml with:

serverless.yml

service: numpy-test

provider:
  name: aws
  runtime: python3.6

functions:
  numpy:
    handler: handler.main

run the following

run:

npm init

[accept defaults, then type "yes"]

npm install --save serverless-python-requirements

now replace serverless.yml with:

serverless.yml

service: numpy-test

provider:
  name: aws
  runtime: python3.6

functions:
  numpy:
    handler: handler.main

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

package:
  exclude:
    - venv/**

Run the following

Run

serverless deploy --aws-profile [aws-account-you-want-to-upload-fxn-to]
serverless invoke -f numpy --log

If all goes well, you should see something like this:

Expected output:

START RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4 Version: $LATEST
Your numpy array:
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]
END RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4
REPORT RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4  Duration: 0.52 ms    
Billed Duration: 100 ms     Memory Size: 1024 MB    Max Memory Used: 37 MB

Tchotchke · Accepted Answer · 2017-03-01 15:49:04Z

0

Building on @MarkB's comment, it would not be possible to build numpy on a Mac to use on AWS Lambda. So why do some packages work and others don't?

Python extension modules, as explained on Mark Nunnikhoven's blog here, are

written in C or C++ that can either extend python or call C or C++ libraries.

Since these modules are compiled specific to the system you're on, and AWS Lambda is a Linux environment, you'll need to install any extension modules on a Linux environment.

answered Mar 1, 2017 at 15:49

Tchotchke

3,1213 gold badges27 silver badges38 bronze badges

Comments

adarsh vemali · Accepted Answer · 2021-05-14 12:09:39Z

-1

For using other packages which aren't available in AWS layers follow the following steps.

Go to your terminal

Make a directory named "python".

Go into the directory.

Enter "pip install " command.

Zip this file and add the zipped file to the AWS

Follow this link for more details : https://www.youtube.com/watch?v=3BH79Uciw5w

answered May 14, 2021 at 12:09

adarsh vemali

11 bronze badge

Collectives™ on Stack Overflow

Installing numpy on Mac to work on AWS Lambda

4 Answers 4

1 Comment

Create a Lambda Function that uses numpy

Run the following:

handler.py

Run the following:

serverless.yml

run:

serverless.yml

Run

Expected output:

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Create a Lambda Function that uses numpy

Run the following:

handler.py

Run the following:

serverless.yml

run:

serverless.yml

Run

Expected output:

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related