AWS Lambda (Python) - Downloading file from internet and upload directly to AWS S3

Question

I am trying to download a file from an API to upload (stream) directly into S3.

My code for local downloads (which works perfectly):

import requests
import datetime
import os

headers = {'Authorization': 'apikey THISISHIDDEN'}
baseURL = 'https://api.test.au/busschedule/'
target_path = datetime.datetime.now().strftime('%Y-%m-%d schedule') + '.zip'

response = requests.get(baseURL, stream=True, headers=headers)
handle = open(target_path, "wb")
for chunk in response.iter_content(chunk_size=512):
    if chunk:  # filter out keep-alive new chunks
        handle.write(chunk)
handle.close()

My attempt to download and stream to S3 (which didn't work):

# import requests
import datetime
import os
import boto3
import botocore.vendored.requests.packages.urllib3 as urllib3

# Get environment variables from serverless.yml
bucket = "bucket"  
s3folder = "schedules"

# Set standard script parameters
headers = {'Authorization': 'apikey THISISHIDDEN'}
baseURL = 'https://api.test.au/busschedule/'


def run(event, context):
    s3 = boto3.client('s3')
    datetimestamp = datetime.datetime.today().strftime('%Y%m%dT%H%M%S')
    filename = datetimestamp + " bus schedule.zip"
    key = s3folder + '/' + filename  # your desired s3 path or filename
    http = urllib3.PoolManager()
    s3.upload_fileobj(http.request('GET', baseURL,
                                   headers=headers, preload_content=False),
                                   bucket, key)


def main():
  run({},{})


if __name__ == "__main__":
        exit(main())

The error I get returned by CloudWatch is:

InsecureRequestWarning: Unverified HTTPS request is being made.  Timeout after 300.10s.

EDIT: The lambda function has a timeout of 300 seconds; but this should be more than long enough to download the file (6mb). Downloading locally is done within 10 or so seconds. Does anyone have a better approach to this?

This error means that this unhandled exception was raised during initialization (when your code is imported). Probably some code tries to get os.environ['URL']. Any chance something missing in the code here? BTW this code is malformatted, you have an invalid string literal — Ronyis
– Ronyis, Commented Mar 31, 2019 at 14:32
My apologies, in an attempt to clean up the code for Stack Overflow, I've uploaded the wrong issue. I am using serverless to deploy the code to AWS Lambda and misnamed the environment variable (URL). — dandev91
– dandev91, Commented Mar 31, 2019 at 15:40
What is the memory size? Using low memory size will also mean low network bandwith — Ronyis
– Ronyis, Commented Mar 31, 2019 at 16:34
512mb memory allocated for a very small (less than 10mb) file — dandev91
– dandev91, Commented Mar 31, 2019 at 17:00
I suggest to first run the GET request, then the S3 upload, and add prints to see where the timeout occur. You can also try with a fetching a smaller file size to see if it's somehow related. — Ronyis
– Ronyis, Commented Mar 31, 2019 at 17:03

dandev91 · Accepted Answer · 2019-04-02 06:24:57Z

1

Resolved this issue using the 'smart_open' library:

response = requests.get(baseURL, stream=True, headers=headers)
s3url = 's3://' + bucket + '/' + key
with smart_open(s3url, 'wb') as fout:
    fout.write(response.content)

I have another issue to resolve (Lambda permissions) but this will be a separate question. Running this locally worked a treat.

answered Apr 2, 2019 at 6:24

dandev91

1,7415 gold badges24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

bhave7 · Accepted Answer · 2022-12-05 05:34:59Z

0

Another way of uploading file (even larger than 6MB) using AWS lambda:

Step 1: Create a pre-signed URL based on the get or put request, return this url as the response. Step 2: Use this URL in your file uploader class on UI.

Pros

Code is scalable in terms of file size since lambda has limit of 6MB, this logic works with lesser and higher data.
Pre-Signed URL is secure way, we can provide the time limit for the url after which the url is expired.

Cons

Its a 2 step process, so need to hit 2 APIs to upload single file.

Please feel free to give feedback for this approach.

answered Dec 5, 2022 at 5:34

bhave7

1061 silver badge4 bronze badges

Collectives™ on Stack Overflow

AWS Lambda (Python) - Downloading file from internet and upload directly to AWS S3

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related