AWS Lambda unable to GET a file from S3

Question

Hi I have a lambda (python3.6) below that is unable to read a file from S3, even though the lambda is in a role that has unfettered permissions for S3 (IAM policy below).

The Lambda simply attempts to retrieve a file from S3 and write it to a temporary location. However it blocks on calling s3.Bucket() and times out (even with a timeout in the minutes).

What's really weird is that it's timing out without any exception, and not rejecting the call to s3.Bucket() with some kind of permission error.

This is pretty basic, but I'm at a loss to get this sorted out.

import boto3
from botocore import exceptions

def lambda_handler(event, context):
  key = event['image']
  bucket = event['bucket']
  tempfile = '/tmp/%s-%s' % (bucket, key)
  print('(p) bucket: %s::%s' % (bucket, key))
  print('(p) tempfile: %s' % tempfile)
  s3 = boto3.resource('s3')
  print('(p) resource intiialized')
  try:
    b = s3.Bucket(bucket) 
    print('(p) bucket info: %s [%s]' % (b.name, b.creation_date))
    b.download_file(prefixed_key, tempfile)
    print('(p} file downloaded to %s' % tempfile)
  except exceptions.ParamValidationError as e:
    return {"statusCode": 400, "body": 'ParamValidationError: [%s]' % e}
  except exceptions.ClientError as e:
    message = '[%s]: [%s]' % (e.response['Error']['Code'], e.response['Error']['Message'])
    return {"statusCode": e.response['ResponseMetadata']['HTTPStatusCode'], "body": message}
  print('(p) image downloaded from s3 and stored at: %s' % tempfile)
  return None

IAM Policy that the role has is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my_bucket",
                "arn:aws:s3:::my_bucket/*"
            ]
        }
    ]
}

Example logs:

22:42:43
START RequestId: 61c60167-839d-11e7-97b1-a772bbde2609 Version: $LATEST
START RequestId: 61c60167-839d-11e7-97b1-a772bbde2609 Version: $LATEST
22:42:43
(p) bucket: my_bucket::my_key
22:42:43
(p) tempfile: /tmp/my_bucket/my_key
22:42:43
(p) resource intiialized
22:43:13
END RequestId: 61c60167-839d-11e7-97b1-a772bbde2609
END RequestId: 61c60167-839d-11e7-97b1-a772bbde2609

Do you mean the Lambda function itself is timing out? Did you increase the function timeout from the default of 3 seconds? Is it running in a VPC with no outbound network connectivity? — jarmod
– jarmod, Commented Aug 17, 2017 at 22:46
I've set the function timeout to be as much as 2 minutes, and the Lambda itself times out. I've added some log output to illustrate. — Edward Q. Bridges
– Edward Q. Bridges, Commented Aug 17, 2017 at 22:50
What are the details of the exception messages? You're not getting the "image downloaded" message, so your function must be returning before that -- and the only returns in that sequence are in the exception clauses. — Chris Johnson
– Chris Johnson, Commented Aug 17, 2017 at 22:58
There are no exception messages. Even if the timeout setting is in the minutes, it neither returns nor completes execution. — Edward Q. Bridges
– Edward Q. Bridges, Commented Aug 17, 2017 at 23:09

Edward Q. Bridges · Accepted Answer · 2017-08-18 00:20:49Z

0

The issue was narrowed down to a misconfiguration of the VPC. I simply configured it to run outside of a VPC as I don't need that for now, and it worked.

answered Aug 18, 2017 at 0:20

Edward Q. Bridges

18.6k8 gold badges37 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hum3 · Accepted Answer · 2019-03-02 10:10:57Z

0

I had a similar problem and I had not installed a VPC gateway endpoint. Although my own instances in my subnets in the VPC could access the S3 URLS, the Lambda instances could not. This was fixed by adding a VPC gateway with default options to allow my subnets to use it. Now my Lambda instances are able to access S3.
This is a free option as compared to using a NAT gateway.

PS you need to make permissions were granted on S3 object. I made it public readable which is probably a bit too wide.

answered Mar 2, 2019 at 10:10

hum3

1,7711 gold badge18 silver badges22 bronze badges

Collectives™ on Stack Overflow

AWS Lambda unable to GET a file from S3

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related