How to load JSON data (call from API) without key directly to S3 bucket using Python?

Question

I am relatively new to AWS s3 I am calling an API to load the JSON data directly to s3 bucket. From s3 bucket data will be read by Snowflake. After researching I found that using Boto3 we can load data into s3 directly. Code will look something like below, however one thing I am not sure about is What should I put for the key as there is no object created in my S3 bucket. Also, what is the good practice to load the JSON data to s3 ? Do I need to encode JSON data to 'UTF-8' as done here by SO user Uwe Bretschneider.

Thanks in advance!

Python code:

import json,urllib.request
import boto3
data = urllib.request.urlopen("https://api.github.com/users?since=100").read()
output = json.loads(data) 
print (output) #Checking the data


s3 = boto3.client('s3')
s3.put_object(
     Body=str(json.dumps(data))
     Bucket='I_HAVE_BUCKET_NAME'
     Key='your_key_here'
)

Key is analogous to path. It's just a name you use to reference the data in the bucket, it can be whatever you like and you don't need to create a parent path in S3. So you could specify 'a/b/c/d.json' without 'a/b/c' previously existing. — NullPointerException
– NullPointerException, Commented May 21, 2022 at 15:52

Greg · Accepted Answer · 2022-05-21 16:00:57Z

3

By using put_object, which means you are creating a new object in the bucket, so there is no existing key. This key is just like a file name in the file system. You can specify whatever names you like, such as my-data.json or some-dir/my-data.json. You can find out more in https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.

As for encoder, it's always good to specify the encoding IMO, just to make sure your source file has properly encoded too.

answered May 21, 2022 at 16:00

Greg

1,62315 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

biggboss2019 Over a year ago

Thank you! Where should I put encoder in my above code ? I am receiving "TypeError: Object of type 'bytes' is not JSON serializable"

Greg Over a year ago

Played with the code for a while, think you can use output directly: output = json.loads(data.decode('utf-8')) ... Body=str(json.dumps(output )) see if that helps, could ignore my previous comment.

Collectives™ on Stack Overflow

How to load JSON data (call from API) without key directly to S3 bucket using Python?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related