Why does my Lambda function write an empty csv file to S3?

Question

I'm calling the YouTube API to download and store channel statistics in S3. I can write a csv file to my S3 bucket without any errors, but it's empty. I have checked this thread Why the csv file in S3 is empty after loading from Lambda, but I'm not using a with block in to_csv_channel().

I'm currently running the script locally with Windows task scheduler with a slightly modified to_csv_channel() method that appends data to an existing csv, but in this case I would like to write a new csv file to S3 every day.

import json
import requests
from datetime import date
from datetime import datetime
import csv
import boto3

class YTstats:

    def __init__(self, api_key, channel_id):
        self.api_key = api_key
        self.channel_id = channel_id
        self.channel_name = None
        self.channel_statistics = None

    def get_channel_title(self):
        """Get the channel name."""

        url_title = f'https://www.googleapis.com/youtube/v3/channels?part=snippet&id={self.channel_id}&key={self.api_key}'

        json_url_title = requests.get(url_title)
        channel_name_json = json.loads(json_url_title.text)
        channel_name_json = channel_name_json['items'][0]['snippet']['title']

        self.channel_name = channel_name_json
        return channel_name_json

    def get_channel_statistics(self):
        """Extract channel statistics"""

        url = f'https://www.googleapis.com/youtube/v3/channels?part=statistics&id={self.channel_id}&key={self.api_key}'

        json_url = requests.get(url)
        statistics = json.loads(json_url.text)
        statistics = statistics['items'][0]

        self.channel_statistics = statistics
        return statistics

    def to_csv_channel(self):
        """Saves the channel statistics to a csv file."""
        
        s3 = boto3.client('s3')

        date_col = datetime.now()
        dt = datetime.strftime(date_col, '%Y-%m-%d')

        self.get_channel_title()
        channel = self.channel_name
        channel_id = self.channel_statistics['id']
        views = self.channel_statistics['statistics']['viewCount']
    
        temp_csv_file = csv.writer(open('/tmp/youtube.csv', 'w'))
        temp_csv_file.writerow(['channel_id', 'channel', 'views']) # column headers 
        temp_csv_file.writerow({'channel_id': channel_id, 'channel': channel, 'views': views}) # rows

        final_file_name='youtube-api/youtube_'+ dt +'.csv'
        
        s3.upload_file('/tmp/youtube.csv', Bucket = 'my-aws-bucket', Key = final_file_name)

My lambda_handler:

from youtube import YTstats

def lambda_handler(event, context):

    key = 'my_youtube_api_key'

    # Mike Shake, Studson Studio
    channel_ids = ['UC6ktP3PLU5sAJxN9Rb0TALg','UC6u6uY4VbvuNtU0BU7F9olw']

    for id in channel_ids:
        yt = YTstats(key, id)
        yt.get_channel_statistics()
        yt.to_csv_channel()

    return {
        'statusCode': 200
    }

Any help would be appreciated.

take a look on this post How to write a file or data to an S3 object using boto3 — samtoddler
– samtoddler, Commented Mar 6, 2021 at 20:53

samtoddler · Accepted Answer · 2021-03-06 21:43:45Z

0

import boto3

def lambda_handler(event, context):
    bucket_name = 'mybucketname'
    object_key =  'folder/sub/path/to/s3key'
    s3 = boto3.resource('s3')
    file = open("/tmp/myfile.txt", "w") 
    file.write("Your text goes here") 
    file.close()
    s3.Bucket(bucket_name).upload_file('/tmp/myfile.txt', object_key)

S3 Service Resource

similar post : How to write a file or data to an S3 object using boto3

answered Mar 6, 2021 at 21:43

samtoddler

9,8533 gold badges28 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why does my Lambda function write an empty csv file to S3?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related