4

How can I use pymongo in a lambda function?

After following the instructions from AWS - https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-dependencies, I've found that the pymongo library is not recognized by my lambda handler function, but other local packages are.

The python-dateutil is a local package that works as expected , however the pymongo package is not recognized as an import and fails when run.

My file name is correctly listed in the lambda handler function as well (hotel-car-rental.lambda_handler) and other local package imports work fine. So I believe it is something to do with pymongo? How can I get pymongo to work with aws lambda?

Here is the lambda file structure:

  • BookTrip
    • dateutil
    • pymongo
    • hotel-car-rental.py

Here is the code:

import json
import datetime
import dateutil.parser  # <--- Works with local package
import logging

from pymongo import MongoClient   # <--- Does NOT work with local package

# --- Main handler ---

def lambda_handler(event, context):
    """
    Route the incoming request based on intent.
    The JSON body of the request is provided in the event slot.
    """

    logger.debug('event.bot.name={}'.format(event['bot']['name']))

    return dispatch(event)

3 Answers 3

1

The way I solved this issue was by packaging up the entire 'site-packages' directory in my virtual environment, rather than just the individual package locations. It appears pymongo installs additional dependencies, such as bson library, that are required.

Here is the specific process I used to package installed libraries for use in aws lambda.

mkdir -p ./package/tmp/lib
cp -a ./lambda/. ./package/tmp/
cp -a venv/lib/python3.7/site-packages/. ./package/tmp/
rm -rf ./package/tmp/wheel* && rm -rf ./package/tmp/pip*
cd ./package/tmp && zip -r ../../hotel-car-rental.zip .
Sign up to request clarification or add additional context in comments.

Comments

1

You could also - instead of doing this since it requires copying/tinkering with your own stuff - run this inside a docker container to package up lambda.

docker run -v <code directory>:/function -it --rm ubuntu
apt-get update
apt-get install python-pip
apt-get install zip

cd function
pip install -t . pymongo
cd ..
zip -r linux-lambda.zip working

The only reason I do this now is that when using a Mac - which I do - it can sometimes give an "Invalid ELF Header" error and Lambda won't compile!

Comments

0

I'm building AWS lambdas via the Serverless framework and had a similar problem. However, I've found an official example and noticed that they provide a specific plugin for that called serverless-python-requirements. Once I've added it to serverless.yml

plugins:
  - serverless-python-requirements

I've got a warning (on serverless deploy) that I have to install the plugin, which is done via

serverless plugin install -n serverless-python-requirements

Once I've done that, import from pymongo no longer causes the problem.
(also I had to add node_modules/ to .gitignore and add package files to git)


Note: there's also a known problem that some users including those using Windows + VS Code/Powershell may face with serverless-python-requirements: if one creates requirements.txt via

pip freeze > requirements.txt

that may result in wrong encoding and serverless deploy fails in that case. Changing encoding to UTF-8 fixes the issue. They suggest that using

pip freeze | Out-File -Encoding UTF8 requirements.txt

instead allows to avoid this problem.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.