AWS sagemaker error - AttributeError: 'NoneType' object has no attribute 'startswith'

Question

As per this - How to use a pretrained model from s3 to predict some data? , I was trying to use an existing model to create an endpoint, but I was facing the following error -

    Traceback (most recent call last):
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
    super().handle_request(listener_name, req, sock, addr)
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving.py", line 124, in main
    serving_env.module_dir)
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving.py", line 101, in import_module
    user_module = importlib.import_module(module_name)
  File "/miniconda3/lib/python3.7/importlib/__init__.py", line 118, in import_module
    if name.startswith('.'):

As per Problem deploying the best estimator gotten with sagemaker.estimator.Estimator (w/ sklearn custom image), https://forums.aws.amazon.com/thread.jspa?threadID=313838 , I am using the correct env variables (along with SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT, SAGEMAKER_PROGRAM, and SAGEMAKER_SUBMIT_DIRECTORY), but somehow the health checks are failing while creation of the endpoint.

I tried the similar thing via AWS console and it is working surprisingly. Is there a work around for this to do it via code?

My code snippet:

trainedmodel = sagemaker.model.Model(
model_data='s3://my-bucket/my-key/output/model.tar.gz',
image='my-image',
env={"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv", 
     "SAGEMAKER_USE_NGINX": "True", 
     "SAGEMAKER_WORKER_CLASS_TYPE": "gevent", 
     "SAGEMAKER_KEEP_ALIVE_SEC": "60", 
     "SAGEMAKER_CONTAINER_LOG_LEVEL": "20",
     "SAGEMAKER_ENABLE_CLOUDWATCH_METRICS": "false",
     "SAGEMAKER_PROGRAM": "my-script.py",
     "SAGEMAKER_REGION": "us-east-1",
     "SAGEMAKER_SUBMIT_DIRECTORY": "s3://my-bucket/my-key/source/sourcedir.tar.gz"
    },
role=role)

trainedmodel.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge', endpoint_name = 'my-endpoint')

Delta Charlie · Accepted Answer · 2021-11-17 10:53:05Z

6

For anyone else landing here, I had a similar issue when creating sklearn models from a ModelPackage.

Error message in endpoint logs when trying to create endpoint:

AttributeError: 'NoneType' object has no attribute 'startswith'

Solved with the following when defining the model package:

Env variable SAGEMAKER_SUBMIT_DIRECTORY should be set to the directory on the container, normally '/opt/ml/model/'
SAGEMAKER_PROGRAM should be set to the name of the serving script, e.g. 'sagemaker_serve.py'

These are specified under the 'Environment' section part of each entry in the 'Containers' section.

edited Nov 17, 2021 at 10:53

answered Nov 17, 2021 at 10:46

Delta Charlie

1111 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ritesh Over a year ago

In case anyone would like to know the details regarding code implementation, kindly refer https://github.com/RamVegiraju/Pre-Trained-Sklearn-SageMaker/blob/master/main.py

Benjamin Zach · Accepted Answer · 2021-11-17 22:29:55Z

0

Based on your stack trace, it looks like the container cannot find your entry_point module (my-script.py).

By default, the container will add 'opt/ml/code' to the Python path and modules under this directory can be imported.

You can modify this path to other values by providing values to SAGEMAKER_BASE_PATH (default to '/opt/ml') and put your script under '<SAGEMAKER_BASE_PATH>/code' and the container will import module 'SAGEMAKER_BASE_PATH>/code/SAGEMAKER_PROGRAM'

edited Nov 17, 2021 at 22:29

Benjamin Zach

1,6412 gold badges21 silver badges43 bronze badges

answered Aug 5, 2020 at 5:35

Deng Chuyang

3322 silver badges3 bronze badges

Comments

Yash Choubey · Accepted Answer · 2021-11-19 22:01:38Z

0

Try using: sagemaker.model.FrameworkModel to create model with required scripts

answered Nov 19, 2021 at 22:01

Yash Choubey

294 bronze badges

Comments

Bruno · Accepted Answer · 2022-02-09 20:29:33Z

What worked for me was creating the model from a training job and then using the env variables listed in the question:

training_src_file     = "s3://bucket_data_assets/training-src-files.tar.gz"
churn_model = sagemaker_session.create_model_from_job(
    training_job_name=completed_training_job_name, 
    name=None, 
    role=role, image_uri=None, 
    model_data_url=None, 
    env={"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv", 
     "SAGEMAKER_USE_NGINX": "True", 
     "SAGEMAKER_WORKER_CLASS_TYPE": "gevent", 
     "SAGEMAKER_KEEP_ALIVE_SEC": "60", 
     "SAGEMAKER_CONTAINER_LOG_LEVEL": "20",
     "SAGEMAKER_ENABLE_CLOUDWATCH_METRICS": "false",
     "SAGEMAKER_PROGRAM": "train.py", #the entry point present in training-src-files.tar.gz
     "SAGEMAKER_REGION": "us-east-1",
     "SAGEMAKER_SUBMIT_DIRECTORY": training_src_file,
    }

Collectives™ on Stack Overflow

AWS sagemaker error - AttributeError: 'NoneType' object has no attribute 'startswith'

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related