Why i can't use my custom scikit learning model in Lambda AWS?

Question

I have a custom scikit-learning model very simple, it works locally but fails when i try to run it in lambda aws.

The error is the folowing: { "errorMessage": "Can't get attribute 'Modelito' on <module 'main' from '/var/runtime/bootstrap.py'>", "errorType": "AttributeError", "requestId": "c1d83b5f-fcaa-4877-9267-255e07c0bb9e", "stackTrace": [ " File "/var/task/lambda_function.py", line 165, in handler\n model = pickle.load(file)\n" ] }

I have the lambda function and inside of it i paste the custom class and then i try to unpickle the model but it fails! The same setting works locally. I try many diferent things but maybe i'm missing something. Some idea?

ParisNakitaKejser · Accepted Answer · 2024-01-14 10:38:20Z

0

How did your application do, Lambda did not have any "disk" to store the data on so you need to add EBS or EFS for that or load it in your memory from S3 storage.

The next thing is to remember Lambda did not use GPU, so it required the use of CPU and you should add enough memory to scale the CPU threads up to get more CPU power.

I will recommend using a Docker image and running your Lambda with this image from ECR then you can be sure what you do locally is the same inside your Lambda function.

But can you please share your application code Otherwise, it's hard for everyone to help you.

Why will you run sci-kit on Lambda and not as a Batch to EC2 instance (cheaper) in the long term, you can use a Lambda function to trigger this Batch if it's something happens with your event-driving architecture.

answered Jan 14, 2024 at 10:38

ParisNakitaKejser

15.6k9 gold badges51 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why i can't use my custom scikit learning model in Lambda AWS?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related