Fail to get csv from S3 and convert it with Python

Question

I need to read csv file from s3 bucket and insert each row on dynamoDB

def load_users_dynamodb():

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table("test")

obj = s3.get_object(Bucket='test-app-config', Key='extract_Users.csv')
#return obj
data = obj['Body'].read().split('\n')

#return json.dumps(data)

with table.batch_writer() as batch:
    for row in data:
        batch.put_item(Item={
            'registration': row.split(',')[0],
            'name': row.split(',')[1],
            'role': row.split(',')[2],
            'company': row.split(',')[3],
            'hiredcompany': row.split(',')[4],
            'region': row.split(',')[5]
        })

return 'OK'

im getting exception and I can't proceed:

   Response:
{
  "errorMessage": "a bytes-like object is required, not 'str'",
  "errorType": "TypeError",
  "stackTrace": [
  "  File \"/var/task/lambda_function.py\", line 10, in          lambda_handler\n    'body': load_users_dynamodb()\n",
"  File \"/var/task/lambda_function.py\", line 21, in load_users_dynamodb\n    data = obj['Body'].read().split('\\n')\n"
]
}

Someone can help me please? o/

Amit Baranes · Accepted Answer · 2020-05-24 07:14:43Z

4

Your issue related to decoding the object return from s3.You need to read the file as csv.

Take a look at the following code snippet:

    import boto3
    import csv

    s3 = boto3.client('s3')

    def lambda_handler(event, context):
        obj = s3.get_object(Bucket='Bucket_Name', Key='File_Name.csv')
        data = obj['Body'].read().decode('utf-8').splitlines()
        lines = csv.reader(data)
        headers = next(lines)
        print('headers: %s' %(headers))
        for line in lines:
            print(line)

Output :

Dummy csv.

edited May 24, 2020 at 7:14

answered Oct 28, 2019 at 20:28

Amit Baranes

8,1922 gold badges39 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Flávio Costa Over a year ago

"errorMessage": "iterator should return strings, not bytes (did you open the file in text mode?)", "errorType": "Error", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 11, in lambda_handler\n 'body': load_users_dynamodb()\n", " File \"/var/task/lambda_function.py\", line 26, in load_users_dynamodb\n for line in lines:\n" ]

Amit Baranes Over a year ago

Which python version are you using?

Amit Baranes Over a year ago

I guess you work with python 3 while my first example was with python 2. I updated my answer.

Flávio Costa Over a year ago

Sorry about the late and thanks for your solution. o/

Collectives™ on Stack Overflow

Fail to get csv from S3 and convert it with Python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related