moving local data to google cloud bucket using python api

Question

I can move data in google storage to buckets using the following:

gsutil cp afile.txt gs://my-bucket

How to do the same using the python api library:

from google.cloud import storage
storage_client = storage.Client()

# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)

Cant find anything more than the above.

Hi Mpizos, do you mean upload data to a cloud storage bucket (e.g. from your local machine)? — Paul
– Paul, Commented Jun 18, 2018 at 13:13
@Paul yes. Not necessarily from local machine. But generally how to move data to a cloud storage bucket with the python api — Mpizos Dimitris
– Mpizos Dimitris, Commented Jun 18, 2018 at 15:59

Paul · Accepted Answer · 2018-06-18 13:10:26Z

4

There is an API Client Library code sample code here. My code typically looks like below which is a slight variant on the code they provide:

from google.cloud import storage

client = storage.Client(project='<myprojectname>')
mybucket = storage.bucket.Bucket(client=client, name='mybucket')
mydatapath = 'C:\whatever\something' + '\\'   #etc
blob = mybucket.blob('afile.txt')
blob.upload_from_filename(mydatapath + 'afile.txt')

In case it is of interest, another method is to run the "gsutil" command line how you have typed in your Original Post using the subprocess command, e.g.:

import subprocess
subprocess.call("gsutil cp afile.txt gs://mybucket/", shell=True)

In my view, there are pros and cons of both methods depending on what you are trying to achieve - the latter method allows multi-threading if you have many files to upload whereas the former method perhaps allows better control, specification of metadata for each file, etc.

answered Jun 18, 2018 at 13:10

Paul

5587 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Iñigo Over a year ago

I would like to add to Paul's answer two links that may be of help. Cloud Storage Libraries and GIT: Python GCS samples

Dave Liu Over a year ago

shell=True may have security vulnerabilities: stackoverflow.com/questions/3172470/…

Collectives™ on Stack Overflow

moving local data to google cloud bucket using python api

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related