How to copy json file to Amazon S3 using Python

Question

I am experimenting with writing a json file to AWS S3. Below is the sample code. This is the file I want to write 'fileNew.json'. And 'fileOld.json' is an existing file in S3, which I included in the code by mistake and shouldnt be in the code.

df.to_json('fileNew.json', orient='records',lines=True)

os.system('aws s3 cp fileNew.json s3://sbx-myproject/fileOld.json --sse')

Will the above command replace the existing file ? OR will it be just unsuccessful in creating the new file ?

if you're using Python you should probably be using the boto3 library... just sayin' — Wayne Werner
– Wayne Werner, Commented May 11, 2018 at 22:10
It will replace the existing file. S3 doesn't care about the filename fileNew.json. — Alex Hall
– Alex Hall, Commented May 11, 2018 at 22:18

Tom · Accepted Answer · 2021-08-27 02:54:36Z

4

If a file already exists it will be automatically overwritten. So yes fileOld.json will be replaced with the file you are uploading.

While your code will work, it is recommended you use the AWS SDK instead of executing shell commands.

import boto3

data = open('fileNew.json', 'rb')
s3 = boto3.resource('s3')
s3.Bucket('sbx-myproject').put_object(Key='fileOld.json', Body=data)

edited Aug 27, 2021 at 2:54

answered May 11, 2018 at 22:18

Tom

1,6962 gold badges14 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to copy json file to Amazon S3 using Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related