AWS Lambda with Python 3 and virtualenv failed to install dependencies

Question

I'm working on a project in Python3.6 and I use AWS Lambda to implement some functions in python. I have written a buildspec.yml file to "build" and deploy my function from a repository to lambda functions. Here is the code:

version: 0.2
phases:
 install:
   commands:
     - echo "install step"
     - apt-get update
     - apt-get install zip -y
     - apt-get install python3-pip -y
 pre_build:
   commands:
     - echo "pre_build step"
     - pip install --upgrade pip
     - pip install --upgrade awscli
     - pip install --upgrade virtualenv
     # Define directories
     - export HOME_DIR=`pwd`
     - export PREPROCESSING_DIR=$HOME_DIR/preprocessing
     - export COMPARE_DIR=$HOME_DIR/compareHilightGood
     - export LAUNCH_HILIGHT_DIR=$HOME_DIR/LaunchHiLight
     - export NLTK_DATA=$HOME_DIR/nltk_data
     - mkdir nltk_data
     # create virtual environements
     - cd $HOME_DIR
     - virtualenv venv_preprocessing
     - virtualenv venv_compare
     - export SITE_PACKAGE_PREPROCESSING=$HOME_DIR/venv_preprocessing/lib/python3.6/site-packages
     - export SITE_PACKAGE_COMPARE=$HOME_DIR/venv_compare/lib/python3.6/site-packages
 build:
   commands:
     - echo "build step"
     - cd $HOME_DIR
     # Configure preprocessing virtual environement
     - . venv_preprocessing/bin/activate
       pip install requests
       pip install nltk
       python -m nltk.downloader -d $NLTK_DATA wordnet stopwords punkt
       deactivate
     - mv $NLTK_DATA $SITE_PACKAGE_PREPROCESSING
     - mv $PREPROCESSING_DIR/* $SITE_PACKAGE_PREPROCESSING
     - cd $SITE_PACKAGE_PREPROCESSING
     - sudo zip -r9 -q $HOME_DIR/preprocessing.zip .
     # Configure compare virtual environement
     - cd $HOME_DIR
     - . venv_compare/bin/activate
       pip install gensim
       pip install pandas
       deactivate
     - mv $COMPARE_DIR/* $SITE_PACKAGE_COMPARE
     - cd $SITE_PACKAGE_COMPARE
     - sudo zip -r9 -q $HOME_DIR/compare.zip .
     # Launch hilight
     - cd $LAUNCH_HILIGHT_DIR
     - sudo zip -r9 -q $HOME_DIR/launchHilight.zip .
 post_build:
   commands:
     - echo "post_build step"
     - cd $HOME_DIR
     - ls
     # preprocessing
     - aws s3 rm s3://lambda-preprocessing --recursive
     - aws s3 cp --acl public-read preprocessing.zip s3://lambda-preprocessing/preprocessing.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --s3-bucket lambda-preprocessing --s3-key preprocessing.zip
     - aws lambda update-function-configuration --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --environment 'Variables={NLTK_DATA=/var/task/nltk_data}'
     # compare hilight good
     - aws s3 rm s3://lambda-comparehilightgood --recursive
     - aws s3 cp --acl public-read compare.zip s3://lambda-comparehilightgood/compare.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:compareHilightGood --s3-bucket lambda-comparehilightgood --s3-key compare.zip
     # launchHilight
     - aws s3 rm s3://hilightalgo --recursive
     - aws s3 cp --quiet --acl public-read launchHilight.zip s3://hilightalgo/launchHilight.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:LaunchHilight --s3-bucket hilightalgo --s3-key launchHilight.zip
artifacts:
  files:
    - '**/*'

In this build process, I create two virtualenv, install my dependencies into them and then I zip my lambda deployment packages composed by:

The site-packages of the virtualenv
The sources

After that, I store my zip packages into S3 buckets and I update the code of the function with the aws cli. Everything seems to work fine but I have two problems:

First, the files seem far too light to me (3.8MB). And when I want to test my lambda functions it's like no modules have been installed. See the error below:

Unable to import module 'lambda_function': No module named 'gensim'

I think the virtualenv doesn't have the modules installed because when I downloaded the .zip files I could see that the __pycache__ folder only contains a small easy_install.cpython-36.pyc.

I don't know what I did wrong, but I think the problem comes from my deployment packages. Does anyone have any ideas?

I don't see you doing pip install gensim or some equivalent of that somewhere? Is it part of some of the packages you already installed if so, which? — gabhijit
– gabhijit, Commented Mar 21, 2019 at 17:37
off course I do it in the build phase of the buildspec.yml file. — Louis Singer
– Louis Singer, Commented Mar 21, 2019 at 17:39
I think in the commands part where you run aws ... you need to do it by setting PYTHONPATH where your both virtual envs are added. Assuming aws is some Python script. In the post_build section — gabhijit
– gabhijit, Commented Mar 22, 2019 at 11:30
I don't understand, according to aws, the deployment package only needs the site-packages folder : docs.aws.amazon.com/fr_fr/lambda/latest/dg/… — Louis Singer
– Louis Singer, Commented Mar 22, 2019 at 12:08

amorphic · Accepted Answer · 2019-06-04 04:26:41Z

1

If you're deploying to AWS Lambda you're better off using a framework such as Serverless or Zappa to package your code and dependencies up into a zip file ready for deployment via S3.

Both work, though I prefer Serverless as it's pluggable, cross-language, seems to be better supported and Just Worked for me. This post gives a good rundown of how to get started.

Once you've got Serverless packaging working, your buildspec.yml becomes very simple: install serverless, run serverless package.

You can also use Serverless to manage the AWS infrastructure for you if you like. But I prefer to do that separately using Terraform.

answered Jun 4, 2019 at 4:26

amorphic

77010 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

noobius · Accepted Answer · 2019-03-21 17:58:05Z

1

Did you zip up the site-packages from both lib and lib64 in your virtualenv? I noticed that some package ended up in one place or other, and I have to package them up from both places.

See if you can find if gensim is installed inside site-packages in either lib/... or lib64/...

answered Mar 21, 2019 at 17:58

noobius

1,5218 silver badges14 bronze badges

1 Comment

Louis Singer Over a year ago

Thanks for the info but my virtualenv doesn't have a lib64 folder: ‘/codebuild/output/src128672841/src/venv_preprocessing/lib64/python3.6/site-packages/*’: No such file or directory

Collectives™ on Stack Overflow

AWS Lambda with Python 3 and virtualenv failed to install dependencies

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related