Access Denied using boto3 through aws Lambda

Question

I use the data processing pipeline constructed of

S3 + SNS + Lambda

becasue S3 can not send notificaiton out of its storage region so I made use of SNS to send S3 notification to Lambda in other region.

The lambda function coded with

from __future__ import print_function
import boto3


def lambda_handler (event, context):
    input_file_bucket = event["Records"][0]["s3"]["bucket"]["name"]
    input_file_key = event["Records"][0]["s3"]["object"]["key"]

    input_file_name = input_file_bucket+"/"+input_file_key

    s3=boto3.resource("s3")
    obj = s3.Object(bucket_name=input_file_bucket, key=input_file_key)
    response = obj.get()

    return event #echo first key valuesdf

when I ran save and test, I got the following error

    {
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      20,
      "lambda_handler",
      "response = obj.get()"
    ],
    [
      "/var/runtime/boto3/resources/factory.py",
      394,
      "do_action",
      "response = action(self, *args, **kwargs)"
    ],
    [
      "/var/runtime/boto3/resources/action.py",
      77,
      "__call__",
      "response = getattr(parent.meta.client, operation_name)(**params)"
    ],
    [
      "/var/runtime/botocore/client.py",
      310,
      "_api_call",
      "return self._make_api_call(operation_name, kwargs)"
    ],
    [
      "/var/runtime/botocore/client.py",
      395,
      "_make_api_call",
      "raise ClientError(parsed_response, operation_name)"
    ]
  ],
  "errorType": "ClientError",
  "errorMessage": "An error occurred (AccessDenied) when calling the GetObject operation: Access Denied"
}

I configured the lambda Role with

full S3 access

and set bucket policy on my target bucket

everyone can do anything(list, delete, etc.)

It seems that I haven't set policy well.

Rob Rose · Accepted Answer · 2018-05-19 18:20:27Z

32

Omuthu's answer actually correctly identified my problem, but it didn't provide a solution so I thought I'd do that.

It's possible that when you setup your permissions in IAM you made something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::test"
            ]
        }
    ]
}

Unfortunately, that's not correct. You need to apply the Object permissions to the objects in the bucket. So it has to look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::test"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::test/*"
            ]
        }
    ]
}

Note the second ARN witht the /* at the end of it.

answered May 19, 2018 at 18:20

Rob Rose

1,98225 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

chaikov Over a year ago

Hey where can I see this settings? I cannot find it anywhere...

Anatolii Stepaniuk Over a year ago

s3:ListBucket is needed to create bucket

Moo Over a year ago

The Amplify CLI set it up the incorrect way when I added a storage trigger. This was a frustrating bug...

Amri · Accepted Answer · 2016-02-01 02:11:09Z

17

I had a similar problem, I solved it by attaching the appropriate policy to my user.

IAM -> Users -> Username -> Permissions -> Attach policy.

Also make sure you add the correct access key and secret access key, you can do so using AmazonCLI.

answered Feb 1, 2016 at 2:11

Amri

1,1109 silver badges16 bronze badges

Comments

omuthu · Accepted Answer · 2020-07-23 12:24:34Z

10

Possibility of the specific S3 object which you are looking for is having limited permissions

S3 object level permission for read is denied
The role attached to lambda does not have permission to get/read S3 objects
If access granted using S3 bucket policy, verify read permissions are provided

edited Jul 23, 2020 at 12:24

answered Nov 7, 2015 at 19:49

omuthu

6,3631 gold badge30 silver badges39 bronze badges

5 Comments

Patrick Perini Over a year ago

This is pretty vague. Could you possibly point out how one might address this issue?

omuthu Over a year ago

Two possibilities 1. S3 object level permission for read is denied 2. The role attached to lambda does not have permission to get/read S3 objects

Vitaly Zdanevich Over a year ago

For me helped adding of s3:GetObject to the policy.

Pedro Over a year ago

I had to write a bucket policy to grant access. I had to add to bucket policy: ` { "Version": "2012-10-17", "Statement": [ { "Sid": "Allow All", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::<userid>:user/<username>" ] }, "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::<resource-name>/*" } ] } `

DrCord Over a year ago

Also worth pointing out that AWS S3 returns Access Denied for objects that don't exist so as to not reveal whether an object exists or not...

Miae Kim · Accepted Answer · 2018-07-26 03:58:25Z

7

I had similar problem, the difference was the bucket was encrypted in KMS key.

Fixed with: IAM -> Encryption keys -> YOUR_AWS_KMS_KEY -> to your policy or account

answered Jul 26, 2018 at 3:58

Miae Kim

1,80120 silver badges22 bronze badges

1 Comment

george Over a year ago

you saved me! :) I need to buy you a coffee!

Tal Joffe · Accepted Answer · 2019-11-26 11:30:07Z

7

Adding to Amri's answer, if your bucket is private and you have the credentials to access it you can use the boto3.client:

import boto3
s3 = boto3.client('s3',aws_access_key_id='ACCESS_KEY',aws_secret_access_key='SECRET_KEY')
response = s3.get_object(Bucket='BUCKET', Key='KEY')

*For this file: s3://bucket/a/b/c/some.text, Bucket is 'bucket' and Key is 'a/b/c/some.text'

---EDIT---

You can easily change the script to accept keys as environment variables for instance so they are not hardcoded. I left it like this for simplicity

edited Nov 26, 2019 at 11:30

answered Mar 14, 2017 at 11:43

Tal Joffe

5,8984 gold badges28 silver badges32 bronze badges

3 Comments

300D7309EF17 Over a year ago

this is a bad idea in a Lambda function. There's no reason to hardcode keys.

Tal Joffe Over a year ago

@tedder42 sometimes it makes sense. let's say you need to copy between buckets with different permissions.

Marlon Teixeira Over a year ago

It has solved my problem! ;)

VIPIN KUMAR · Accepted Answer · 2019-08-04 17:15:49Z

1

In my case - the Lambda which I was running had a role blahblahRole and this blahblahRole didn't have the permission on S3 bucket.

answered Aug 4, 2019 at 17:15

VIPIN KUMAR

3,1572 gold badges25 silver badges37 bronze badges

1 Comment

DJ_Stuffy_K Over a year ago

I have s3* permissions but still I'm getting the error.

Vivek Puurkayastha · Accepted Answer · 2022-02-22 14:51:44Z

1

Following is a simple JSON IAM snippet tha could be added to the IAM Role. Although this gives FullAccess to the bucket, generally should not be a problem because dedicated buckets are used for Lambda operations in production.

But if FullAccess is not desired than whatever operations are throwing AccessDenied Error, those will need to be added separately.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAllOnThisBucket",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": "arn:aws:s3:::bucket_name/*"
        }
    ]
}

answered Feb 22, 2022 at 14:51

Vivek Puurkayastha

5462 gold badges11 silver badges23 bronze badges

Collectives™ on Stack Overflow

Access Denied using boto3 through aws Lambda

7 Answers 7

3 Comments

Comments

5 Comments

1 Comment

3 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

3 Comments

Comments

5 Comments

1 Comment

3 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related