I am using a docker container image on lambda to run my ML model. My lambda function has a S3 trigger to fetch images. I am trying to run my lambda function but I am getting this error. Can someone please help me?
PS - now i am aware /tmp is the only writable directory in lambda but how to solve this with that?
As others have mentioned, /tmp is the only writable directory in any AWS Lambda environments, either using containers or not.
Having said that, you should move your entire library (during the lambda runtime -- during container image build time doesn't work) to that directory -- such that everything remains connected within the library -- and then reference your new library directory in the library path environment for Lambda: LD_LIBRARY_PATH
Referencing your new library directory in the library path environment for Lambda should be done because Lambda looks at the /opt/ directory by default; and since you just moved your library to /tmp, you should also update LD_LIBRARY_PATH to contain that location. This can be done in the Dockerfile:
# Set the LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH="/opt/my-lib-folder/:$LD_LIBRARY_PATH"
or during Lambda runtime:
os.environ['LD_LIBRARY_PATH'] = '/tmp/my-lib-folder:' + os.environ['LD_LIBRARY_PATH']
def lambda_handler(event, context):
# your code ...
If there are still problems, it may be related to linking problems of your library, or that you didn't update your LD_LIBRARY_PATH correctly.
EDIT: As pointed out by @rok, you cannot move your libraries during the container image build time, because the /tmp folder will be erased by AWS automatically.
/tmp folder because never needed them to be writable, but it's good to know that it wouldn't work. I will correct my answer.The file system in a Lambda environment is read-only except for the /tmp directory.
That error seems to be it trying to create its config/cache folders and failing. Essentially you need to specify the MPLCONFIGDIR environment variable to your container and set the value to a directory inside /tmp that is writable.
