1

I made a classifier in Python that uses a lot of libraries. I have uploaded the model to Amazon S3 as a pickle (my_model.pkl). Ideally, every time someone uploads a file to a specific S3 bucket, it should trigger an AWS Lambda that would load the classifier, return predictions and save a few files on an Amazon S3 bucket.

I want to know if it is possible to use a Lambda to execute a Jupyter Notebook in AWS SageMaker. This way I would not have to worry about the dependencies and would generally make the classification more straight forward.

So, is there a way to use an AWS Lambda to execute a Jupyter Notebook?

Improve this question

4 Answers 4

Reset to default
2

Scheduling notebook execution is a bit of a SageMaker anti-pattern, because (1) you would need to manage data I/O (training set, trained model) yourself, (2) you would need to manage metadata tracking yourself, (3) you cannot run on distributed hardware and (4) you cannot use Spot. Instead, it is recommended for scheduled task to leverage the various SageMaker long-running, background job APIs: SageMaker Training, SageMaker Processing or SageMaker Batch Transform (in the case of a batch inference).

That being said, if you still want to schedule a notebook to run, you can do it in a variety of ways:

  • in the SageMaker CICD Reinvent 2018 Video, Notebooks are launched as Cloudformation templates, and their execution is automated via a SageMaker lifecycle configuration.
  • AWS released this blog post to document how to launch Notebooks from within Processing jobs

But again, my recommendation for scheduled tasks would be to remove them from Jupyter, turn them into scripts and run them in SageMaker Training

No matter your choices, all those tasks can be launched as API calls from within a Lambda function, as long as the function role has appropriate permissions

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

I agree with Olivier. Using Sagemaker for Notebook execution might not be the right tool for the job.

Papermill is the framework to run Jupyter Notebooks in this fashion.

You can consider trying this. This allows you to deploy your Jupyter Notebook directly as serverless cloud function and uses Papermill behind the scene.

Disclaimer: I work for Clouderizer.

Improve this answer

Comments

1

This should be the way to go here - https://github.com/aws-samples/sagemaker-run-notebook. In essence, one uses a lambda function to trigger an AWS SageMaker Processing Job. This is kind of like running a notebook on an AWS SageMaker Notebook instance without having to think about provisioning the instance and killing it after finishing the job.

It uses papermill to run the notebook on a sagemaker instance. Users can pass a custom docker image with their own custom kernels.

Improve this answer

Comments

0

It totally possible, not an anti-pattern at all. It really depends on your use-case. AWs actually made a great article describing it, which includes a lambda

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.