Reading data from S3 using Lambda

Question

I have a range of json files stored in an S3 bucket on AWS.

I wish to use AWS lambda python service to parse this json and send the parsed results to an AWS RDS MySQL database.

I have a stable python script for doing the parsing and writing to the database. I need to lambda script to iterate through the json files (when they are added).

Each json file contains a list, simple consisting of results = [content]

In pseudo-code what I want is:

Connect to the S3 bucket (jsondata)
Read the contents of the JSON file (results)
Execute my script for this data (results)

I can list the buckets I have by:

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():
    print(bucket.name)

Giving:

jsondata

But I cannot access this bucket to read its results.

There doesn't appear to be a read or load function.

I wish for something like

for bucket in s3.buckets.all():
   print(bucket.contents)

EDIT

I am misunderstanding something. Rather than reading the file in S3, lambda must download it itself.

From here it seems that you must give lambda a download path, from which it can access the files itself

import libraries

s3_client = boto3.client('s3')

def function to be executed:
   blah blah

def handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key'] 
        download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
        s3_client.download_file(bucket, key, download_path)

ScottMcC · Accepted Answer · 2017-10-03 09:11:38Z

54

s3 = boto3.client('s3')
response = s3.get_object(Bucket=bucket, Key=key)
emailcontent = response['Body'].read().decode('utf-8')

edited Oct 3, 2017 at 9:11

ScottMcC

4,5201 gold badge32 silver badges39 bronze badges

answered Nov 14, 2016 at 18:50

James Hogbin

5411 gold badge4 silver badges2 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ScottMcC Over a year ago

Should also note that you need to create an s3 object to use in your response. i.e. s3 = boto3.client('s3')

Dysosmus · Accepted Answer · 2015-11-18 16:08:49Z

14

You can use bucket.objects.all() to get a list of the all objects in the bucket (you also have alternative methods like filter, page_sizeand limit depending on your need)

These methods return an iterator with S3.ObjectSummary objects in it, from there you can use the method object.get to retrieve the file.

edited Nov 18, 2015 at 16:08

answered Nov 18, 2015 at 16:01

Dysosmus

8429 silver badges15 bronze badges

Collectives™ on Stack Overflow

Reading data from S3 using Lambda

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related