Write and read list of lists with s3 and python

Question

I have a list of lists that I want saved to s3.

Originally, when I saved these locally, I converted them to a csv like this:

from csv import reader, writer

words_list = [['here', 'are', 'some', 'words']['these', 'are', 'in', 'a', 'list']['i', 'love', 'lists']]

# Save list of words list

with open("data/words.csv", "w") as f:
    wr = csv.writer(f)
    wr.writerows(words_list)


# Retrieve list of words list

with open("data/words.csv", 'r') as read_obj:
    csv_reader = reader(read_obj)
    words_list = list(csv_reader)

I am able to save a csv version of the list of lists to s3, I think, using this, which just takes the csv I have saved locally already:

import boto3

session = boto3.Session(
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key
)

s3 = session.resource('s3')

bucket = 'bucket_name'
path = 'data/words.csv'
key = 'data/words.csv'

s3.meta.client.upload_file(key, bucket, path)

However, I've been unable to work out how to read the csv back and convert it to the original list of lists. For example, I've tried this:

# Retrieve from s3
import csv
from io import StringIO 

bucket = 'bucket_name'
path = 'data/words.csv'

client = boto3.client('s3', 
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key
)

csv_obj = client.get_object(Bucket=bucket, Key=path)
body = csv_obj['Body']
csv_string = body.read().decode('utf-8')

But this gets me a string without a sense of where the separate lists start and end, a bit like: "here,are,some,words,these,are,in,a,list,i,love,lists"

I'm not wedded to this approach - I'd be happy with anything that allows me to save the list of lists in a recoverable format in s3. How can I achieve this?

Thanks!

were you able to resolve this ? am stuck with similar kind of requirement. — Ajeet
– Ajeet, Commented Mar 24, 2021 at 9:16

Jaccar · Accepted Answer · 2020-04-21 15:19:53Z

1

What about using json instead of csv?

import json

words_list = [['here', 'are', 'some', 'words'],
              ['these', 'are', 'in','a', 'list'],
              ['i', 'love', 'lists']]

# save to json
with open('list_of_lists.json', mode='w') as fh:
    json.dump(words_list, fh)

# read from json
with open('list_of_lists.json', mode='r') as fh:
    words_list_from_json = json.load(fh)

print(words_list_from_json)

Edited to add:

Then it is possible to read/write the json file to s3.

# Save json file to s3

data = open('list_of_lists.json', 'rb')
bucket = 'bucket_name'
path = "data/list_of_lists.json"

s3.Bucket(bucket).put_object(Key=path, Body=data)

# Retrieve json file from s3

content_object = s3.Object(bucket, path)
file_content = content_object.get()['Body'].read().decode('utf-8')
words_list = json.loads(file_content)

edited Apr 21, 2020 at 15:19

Jaccar

1,87420 silver badges49 bronze badges

answered Apr 21, 2020 at 14:49

Rickey

5241 gold badge4 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jaccar Over a year ago

Thanks for this - I was then able to read/write jsons from s3 so I have added that detail to your answer in an edit.

Collectives™ on Stack Overflow

Write and read list of lists with s3 and python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related