AWS Lambda in Python to copy new files to another s3 bucket

Question

I have created a Lambda which is triggered every time I add a file to s3://test-bucket-01/, it copies all the files in s3://test-bucket-01/ to s3://test-bucket-02/

I would like it to only add the new file that has just been added?

Current Code:

import boto3
s3 = boto3.resource('s3')


def lambda_handler(event, context):
    bucket = s3.Bucket('test-bucket-01')
    dest_bucket = s3.Bucket('tb-bucket-02')
    print(bucket)
    print(dest_bucket)

    for obj in bucket.objects.all():
        dest_key = obj.key
        print(dest_key)
        s3.Object(dest_bucket.name, dest_key).copy_from(CopySource = {'Bucket': obj.bucket_name, 'Key': obj.key})

Results:

Function Logs:
START RequestId: XXXXXXX-XXXXXXX-XXXXXXX Version: $LATEST
s3.Bucket(name='test-bucket-01')
s3.Bucket(name='test-bucket-02')
test-data-01.json
test-data-02.json
test-data-03.json

romruben · Accepted Answer · 2019-05-15 10:33:10Z

You not need to list the bucket to copy all files because the event variable is a json that contains the file that you has been copied into test-bucket-01.

Example:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "C3D13FE58DE4C810",
        "x-amz-id-2": "FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "sourcebucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::mybucket"
        },
        "object": {
          "key": "key/to/HappyFace.jpg",
          "size": 1024,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e"
        }
      }
    }
  ]
}

In this case, this event notify ObjectCreated:Put event. A new file has been copied to "sourcebucket" bucket with key "key/to/HappyFace.jpg".

Amir Shabani · Accepted Answer · 2019-05-16 16:12:53Z

0

Why don't you want to use native functional of AWS S3 to replicate new files from one bucket to another?

It's easier and more reliable. And you can replicate files by prefixes, change storage plans and many others with this native feature.

edited May 16, 2019 at 16:12

Amir Shabani

4,3237 gold badges36 silver badges81 bronze badges

answered May 16, 2019 at 15:37

anton.uspishnyi

3503 silver badges9 bronze badges

2 Comments

J. Patwary Over a year ago

I'm having a similar issue where I'm looking to use lambda instead of s3 native replication. This is because s3 replication isn't able to copy to a different path. e.g. i can't copy from source-bucket/prefix1/ to dest-bucket/prefix2/

user433342 Over a year ago

Because I need instant replication

Anish Gopinath · Accepted Answer · 2020-01-18 16:14:17Z

0

so what is the answer ? how to trigger the lambda only when a new file is added?

answered Jan 18, 2020 at 16:14

Anish Gopinath

1822 gold badges4 silver badges28 bronze badges

Collectives™ on Stack Overflow

AWS Lambda in Python to copy new files to another s3 bucket

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related