Python - Move files between GCP Buckets

Question

I'm trying to move csv files between two existing buckets on GCP. But I'm stuck on an 'Not Found' Error. Here is the code I'm using (This was built after reading GCP docs and SDK's):

from google.cloud import storage


def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name):
    """Moves a blob from one bucket to another with a new name."""
    storage_client = storage.Client()

    source_bucket = storage_client.bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.bucket(destination_bucket_name)

    blob_copy = source_bucket.copy_blob(
        source_blob, destination_bucket, destination_blob_name
    )

    print(
        "Blob {} in bucket {} moved to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    ) 

def list_blobs():
    """Lists all the blobs in the bucket."""

    bucket_name = "BUCKET_1"
    destination_bucket_name = "BUCKET_2/folder"
    
    storage_client = storage.Client()
    
    blobs = storage_client.list_blobs(bucket_name)
    
    for blob in blobs:
        print(blob.name)
        
        destination_blob_name = blob_name = blob.name
        
        move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name)

list_blobs()

And the content from the error message is:

'NotFound: 404 GET https://storage.googleapis.com/storage/v1/b/BUCKET_2/folder?projection=noAcl&prettyPrint=false: Not Found'

Aishwary Shukla · Accepted Answer · 2022-06-03 17:00:00Z

1

Your destination bucket name should just be the bucket name and should not have any prefixes. Just do the following: destination_bucket_name = "BUCKET_2" and add folder/ as a prefix to all blobs being transferred.

answered Jun 3, 2022 at 17:00

Aishwary Shukla

4701 gold badge8 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Renan Saia Over a year ago

Thanks man! It worked! I changed my code with your idea and I finally solved my issue. I changed the code to: blob_name = blob.name destination_blob_name = "folder/{}".format(blob.name)

Andres Fiesco Casasola · Accepted Answer · 2022-06-04 06:01:43Z

Here is a function I found (by dmlee8) that you can use when moving blobs between directories within the same bucket or to a different one:

from google.cloud import storage
import os
    
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="path_to_your_creds.json"
 
def mv_blob(bucket_name, blob_name, new_bucket_name, new_blob_name):
    """
    Function for moving files between directories or buckets . it will use GCP's copy 
    function then delete the blob from the old location.
    
    inputs
    -----
    bucket_name: name of bucket
    blob_name: str, name of file 
        ex. 'data/some_location/file_name'
    new_bucket_name: name of bucket (can be same as original if we're just moving around directories)
    new_blob_name: str, name of file in new directory in target bucket 
        ex. 'data/destination/file_name'
    """
    storage_client = storage.Client()
    source_bucket = storage_client.get_bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.get_bucket(new_bucket_name)
 
    # copy to new destination
    new_blob = source_bucket.copy_blob(
        source_blob, destination_bucket, new_blob_name)
    # delete in old destination
    source_blob.delete()
    
    print(f'File moved from {source_blob} to {new_blob_name}')

Collectives™ on Stack Overflow

Python - Move files between GCP Buckets

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related