SageMaker ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: Could not assume role

Question

When I get to Notebook Instance in SageMaker and create model in linear_learner_mnist example I get error even when I have the role with AmazonSageMakerFullAccessand AssumeRole attached policies attached. When I add AdministratorAccesspolicy to the role - all works fine. What am I missing here?

role exists and attached

Resources:
  SageMakerExecutionRole:
    Condition: RoleArnEmpty
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "sagemaker.amazonaws.com"
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
      Path: "/service-role/"
      Policies:
        Fn::If:
        - S3BucketNotEmpty
        -
          - PolicyName: SageMakerS3BucketAccess
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - s3:*
                  Resource:
                    - !Sub 'arn:aws:s3:::${SageMakerS3Bucket}'
                    - !Sub 'arn:aws:s3:::${SageMakerS3Bucket}/*'
        - Ref: AWS::NoValue

ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: Could not assume role arn:aws:iam::XXX:role/sagemaker-stack-SageMakerExecutionRole-1JT7AT4OPUK9R. Please ensure that the role exists and allows principal 'sagemaker.amazonaws.com' to assume the role.

balast · Accepted Answer · 2019-08-07 00:14:03Z

6

I'm surprised that everything works normally when you add the AdministratorAccess policy. The error and how to correct it is documented at https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html.

An Amazon SageMaker user can grant these permissions with an IAM role (referred to as an execution role). The user passes the role when making these API calls: CreateNotebookInstance, CreateHyperParameterTuningJob, CreateTrainingJob, and CreateModel.

You attach the following trust policy to the IAM role which grants Amazon SageMaker principal permissions to assume the role, and is the same for all of the execution roles:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "sagemaker.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

In the Identity and Access Management (IAM) Console, I added the statement section above to the trust policy under the trust relationships section of my role, and it fixed this error for me.

answered Aug 7, 2019 at 0:14

balast

2092 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

harishkashyap Over a year ago

This still doesn't work at times.

Alohahaha · Accepted Answer · 2019-07-03 20:43:12Z

1

I tried to create an IAM role based on the CloudFormation template you provided and got a role with ARN - arn:aws:iam::xxx:role/service-role/test-iam-SageMakerExecutionRole-187YIQM7UMMXF.

I think the issue is the role ARN you set when creating the training job is not correct. You probably need to add "/service-role" before the role name.

answered Jul 3, 2019 at 20:43

Alohahaha

1062 bronze badges

Collectives™ on Stack Overflow

SageMaker ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: Could not assume role

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related