How to edit S3 files with python

Question

I am working with python and I need to check and edit the content of some files stored in S3.
I need to check if they have a char o string. In that case, I have to replace this char/string.

For example: I want to replace ; with . in following file File1.txt

This is an example;

After replace

File1.txt

This in an example.

Is there a way to do the replace without downloading the file?

Not unless you use something like s3fuse to mount the s3 bucket. github.com/s3fs-fuse/s3fs-fuse — Sathyajith Bhat
– Sathyajith Bhat, Commented Jan 13, 2021 at 16:17
You might also want to consider versioning these objects if you are going to download/edit/upload. — jarmod
– jarmod, Commented Jan 14, 2021 at 0:48

John Rotenstein · Accepted Answer · 2021-01-14 00:28:48Z

7

Objects in Amazon S3 are immutable (they cannot be changed).

I recommend that your code does the following:

Download the file from Amazon S3 to local disk
Perform edits on the local file
Upload the file to Amazon S3 with the same Key (filename)

answered Jan 14, 2021 at 0:28

John Rotenstein

273k28 gold badges456 silver badges541 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

vll1990 · Accepted Answer · 2021-01-14 15:12:16Z

Thanks for the answers.
I need to perform this action in a lambda and this is the result:

    import boto3
    import json
    
    s3 = boto3.client('s3')
    
    def lambda_handler(event, context):
        file='test/data.csv'
        bucket = "my-bucket"
        response = s3.get_object(Bucket=bucket,Key=file )
        
        fileout = 'test/dout.txt'
        rout = s3.get_object(Bucket=bucket,Key=fileout )
    
        
        data = []
        it = response['Body'].iter_lines()
        
        for i, line in enumerate(it):
            # Do the modification here
            modification_in_line = line.decode('utf-8').xxxxxxx  # xxxxxxx is the action
            data.append(modification_in_line)
    
        
        r = s3.put_object(
            Body='\n'.join(data), Bucket=bucket, Key=fileout,)
        
        
        return {
            'statusCode': 200,
            'body': json.dumps(data),
        }

Collectives™ on Stack Overflow

How to edit S3 files with python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related