How can I copy multiple files between buckets using the API

Question

I am using the Google Cloud Storage package for Python to copy a bunch of files from one bucket to another. Basic code is:

from google.cloud.storage.client import Client
def copy_bucket_content (client:Client, source_bucket_name, destination_bucket_name, source_dir):
    source_bucket = client.get_bucket(source_bucket_name)
    destination_bucket = client.get_bucket(destination_bucket_name)
    blobs_to_copy = [blob for blob in source_bucket.list_blobs() if blob.name.startswith(source_dir)]
    source_bucket.
    for blob in blobs_to_copy:
        print ("copying {blob}".format(blob=blob.name))
        source_bucket.copy_blob(blob, destination_bucket, blob.name)

When I pass a source_dir that has many blobs in it the script fails at runtime with:

File "/Users/jamiet/.virtualenvs/hive-data-copy-biEl4iRK/lib/python3.6/site-packages/google/cloud/_http.py", line 293, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.InternalServerError: 500 POST https://www.googleapis.com/storage/v1/b/path/to/blob/copyTo/b/path/to/blob: Backend Error

This invariably occurs after transferring between 50 and 80 blobs (it doesn't fail at the same point each time).

I am assuming that I'm hitting some sort of API request limit. Would that be the case?

If so, how do I get around this? I suppose lifting the restriction would be one way but better would be able to issue just one call to the REST API rather than looping over all the blobs and copying them one at a time. I searched around the GCS python package but didn't find anything that might help.

I assume there's a better way of accomplishing this but I don't know what it is, can anyone help?

F10 · Accepted Answer · 2018-08-10 12:59:20Z

1

There's no quota restriction regarding this scenario. Error 500 indicates a server side issue. You could use an exponential backoff strategy, according to the Handling errors documentation, as well follow the best practices for uploading data.

answered Aug 10, 2018 at 12:59

F10

2,8932 gold badges14 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jamiet Over a year ago

thanks, will give that a go. Any suggestions as to how this can be achieved with a single API call rather than multiple?

F10 Over a year ago

I would use their Transfer Service to move files from one bucket to another

jamiet Over a year ago

I used retrying (pip install retrying) which supports exponential backoff. Worked great. Thanks @F10

Collectives™ on Stack Overflow

How can I copy multiple files between buckets using the API

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related