AWS SDK for node.js not picking up credentials when running in ECS

Question

I have this code:

const uploadAllToS3 = () => {
    console.log("upload useless file to s3, just to test access rights");
    new AWS.S3().putObject({
        Bucket: scheduledJobsConstants.s3BucketName(),
        Key: `test/${new Date()}.txt`, Body: "Hello!",
    }, (err) => {
        console.log(`env:${inspect(process.env)}
        provider:${AWS.CredentialProviderChain.defaultProviders}
        cred:${inspect(AWS.config.credentials)}
        err:${inspect(err)}`.replace(/(\r\n|\n|\r)/gm, ""));
    });
};

When this code runs in ecs(which assumes a role that has access to the S3 bucket), I will get this in the log:

The env part of the log:

env: {
    PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
    HOSTNAME: '485bbd95f87d',
    AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: '/v2/credentials/be3d4d7c-28b6-47e6-8081-cd746d95cb28',
    ECS_CONTAINER_METADATA_FILE: '/opt/ecs/metadata/8c0b731d-49ce-4e19-bf79-64087b433876/ecs-container-metadata.json',
    NODE_VERSION: '8.9.4',
    YARN_VERSION: '1.3.2',
    NPM_CONFIG_PROGRESS: 'false',
    HOME: '/root'
  }

The "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" is present in env var, so I assume the sdk should be able to pick that up.

The provider part of the log is just 4 functions that's defined in the sdk code, the 4th of which would return "ECSCredentials" if it sees "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" in env var.

Strangely the cred part of the log is "null" in ECS. When I run this locally(where I manually assume a role), the cred part of the log is "EnvironmentCredentials", and the file gets uploaded to S3.

The err part of the log is:

{ Forbidden: null 
at Request.extractError (/app/node_modules/aws-sdk/lib/services/s3.js:557:35) 
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:105:20) at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10) 
at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14) 
at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10) 
at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12) 
at /app/node_modules/aws-sdk/lib/state_machine.js:26:10 
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9) 
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12) 
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:115:18) 
message: null, 
code: 'Forbidden', 
region: null, 
time: 2018-01-14T06:30:00.490Z, 
requestId: '4E7121D88BA0FEF1', 
extendedRequestId: 'D5uiXf5EI/OLHeaPhYX67C384ba3SF1I950N78hJWw8Vv4XGQ0opSLOQlXaxVFW31g252dx8YUc=', 
cfId: undefined, 
statusCode: 403, 
retryable: false, 
retryDelay: 118.50320949834776 }

Is there anything wrong with my code or my env var?

Should I print more logs to help diagnose this problem?

Please advise, thanks.

==========Update==========

I have ran this code in ECS const curlCommand = `curl http://169.254.170.2${process.env.AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}`; exec(curlCommand, (error, stdout, stderr) => { console.log(`${curlCommand} err:${error} out:${stdout} stderr:${stderr}`); });

The result I got is: { "SecretAccessKey": "long string", "Token": "very long string", "Expiration": "2018-01-14T08:55:04+0000", "AccessKeyId": "shorted string" }

So if the sdk had called http://169.254.170.2${process.env.AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}, it should have gotten a correct response.

Are you sure that the provider being used is the same on your local and on the ec2 instances ? For example the ec2 instance might have got an invalid cred from the metadata service — Malice
– Malice, Commented Jan 14, 2018 at 8:08
@Malice AWS.config.credentials is null when running in ECS. my assumption is that this should not be null since "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" is present in env var. is that a correct assumption? — Cui Pengfei 崔鹏飞
– Cui Pengfei 崔鹏飞, Commented Jan 14, 2018 at 8:12
You might need to initializa AWS.EC2MetadataCredentials and assign it to aws.config.credentials. As far as I know, it doesnt do this unless it is done explicitly — Malice
– Malice, Commented Jan 14, 2018 at 8:27
@Malice thank you for the doc link. my code is not running in EC2, it's running in ECS containers. according to this: docs.aws.amazon.com/AWSJavaScriptSDK/latest/… it should load ecs credentials if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI exists, which is the case judging by my log — Cui Pengfei 崔鹏飞
– Cui Pengfei 崔鹏飞, Commented Jan 14, 2018 at 8:32

Malice · Accepted Answer · 2018-01-14 09:07:06Z

2

The simple answer is to do

AWS.config.credentials = new AWS.ECSCredentials(options);

presumably because credentials for aws-sdk credentials were not configured/ configured incorrectly.

answered Jan 14, 2018 at 9:07

Malice

1,48218 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Cui Pengfei 崔鹏飞 Over a year ago

Problem solved. It was somehow related to node version. My code was running in ECS, with a base docker image: node:8.9.4. I switched to node:9.4, which is the latest version, that solved the problem. Don't know why, but it's working now.

Aleksi · Accepted Answer · 2020-08-11 10:47:39Z

2

Try using AWS.ECSCredentials.get or getPromise to make sure credentials are loaded before you use them:

const ecsCredentials = new AWS.ECSCredentials({ ... });
await ecsCredentials.getPromise();
AWS.config.credentials = ecsCredentials;

Check this thread for more info: https://github.com/aws/aws-sdk-js/issues/3281#issuecomment-637171993.

answered Aug 11, 2020 at 10:47

Aleksi

5,14639 silver badges51 bronze badges

Comments

redgeoff · Accepted Answer · 2022-11-29 22:56:24Z

1

I am using CloudFormation and in my case, I had to specify TaskRoleArn in the TaskDefinition.Properties before the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI was injected into my container. Once it was, the AWS SDK automatically picked up my credentials.

answered Nov 29, 2022 at 22:56

redgeoff

3,3511 gold badge29 silver badges42 bronze badges

Collectives™ on Stack Overflow

AWS SDK for node.js not picking up credentials when running in ECS

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related