How to read (stream) files from Google Drive in memory in python?

Question

I'm trying to read large CSV files that are dropped on Google Drive using the google-api-python-client https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.MediaIoBaseDownload-class.html

I was able to download the file on the hard drive doing this:

request = drive_service.files().get_media(fileId=file_id)
fh = io.FileIO('test.csv', mode='w')
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
    status, done = downloader.next_chunk()

But I was wondering if there's a simple way to read it in chunks in memory.

Possible duplicate of Python - Download File Using Requests, Directly to Memory — Linda Lawton - DaImTo
– Linda Lawton - DaImTo, Commented Aug 5, 2017 at 13:25

Ahmad Farid · Accepted Answer · 2017-08-18 23:42:30Z

1

    api_service_object = self.service
    request = api_service_object.files().get_media(fileId=file_id)
    stream = io.BytesIO()
    downloader = MediaIoBaseDownload(stream, request)
    done = False
    # Retry if we received HttpError
    for retry in range(0, 5):
        try:
            while done is False:
                status, done = downloader.next_chunk()
                print "Download %d%%." % int(status.progress() * 100)
            return stream.getvalue()
        except HTTPError as error:
            print 'There was an API error: {}. Try # {} failed.'.format(
                error.response,
                retry,
            )

answered Aug 18, 2017 at 23:42

Ahmad Farid

14.8k45 gold badges99 silver badges138 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to read (stream) files from Google Drive in memory in python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related