0

I'm trying to copy an entire folder/blob from an azure adls gen2 container in one subscription to another subscription.

Before the below code I've already connected to azure and populated hash tables of each subscription detail.

Set-AzContext -Subscription $DevDetails.SubscriptionID
$DevStorageAccount = Get-AzStorageAccount -ResourceGroupName $DevDetails.ResourceGroup -StorageAccountName $DevDetails.StorageAccount 
$DevContext = New-AzStorageContext -StorageAccountName $DevDetails.StorageAccount 
$DevStorageContainer = Get-AzStorageContainer -Name "raw" -Context $DevContext

Set-AzContext -Subscription $PPDetails.SubscriptionID
$PPStorageAccount = Get-AzStorageAccount -ResourceGroupName $PPDetails.ResourceGroup -StorageAccountName $PPDetails.StorageAccount 
$PPContext = New-AzStorageContext -StorageAccountName $PPDetails.StorageAccount 
$PPStorageContainer = Get-AzStorageContainer -Name "raw" -Context $PPContext

Set-AzContext -Subscription $ProdDetails.SubscriptionID
$PStorageAccount = Get-AzStorageAccount -ResourceGroupName $ProdDetails.ResourceGroup -StorageAccountName $ProdDetails.StorageAccount 
$PContext = New-AzStorageContext -StorageAccountName $ProdDetails.StorageAccount 
$PStorageContainer = Get-AzStorageContainer -Name "raw" -Context $PContext

I've got the variables for context for all 3 environments.

In the prod storagecontainer raw we have a gen 2 folder in a hierarchical namespace. (Folder1/Folder2/Folder3)

I would like to copy the folder and all contents from prod back to dev. Trying this command.

Start-AzStorageBlobCopy -DestContainer $DevStorageContainer.Name -Context $PContext -SrcBlob "Folder1/Folder2/Folder3/" -SrcContainer $PStorageContainer.Name -DestContext $DevContext

I get the error

Status: 403 (This request is not authorized to perform this operation.)
ErrorCode: CannotVerifyCopySource

I've also tried this.

$prodblob = Get-AzStorageBlob -Context $PContext -Container $PStorageContainer.Name -Blob "Folder1/Folder2/Folder3/"
$prodblob | Start-AzStorageBlobCopy -DestContainer $DevStorageContainer.Name -Context $DevContext

which gives the same error.

I'm able to copy it manually across using azure storage explorer and also able to do it using azcopy so it doesn't seem to be a permission issue.

Is this just not compatible with adls gen2 and I'm going about it completely wrong or is there something simple I'm missing?

3
  • For some reason you can't use azcopy or why did you want to use Start-AzStorageBlobCopy? Commented Dec 7, 2023 at 18:00
  • @KKI mainly because I'm better in powershell and using the command line makes it harder to automatically handle errors. For some reason AZcopy has been repeatedly erroring. I can rerun the command until it succeeds but. Commented Dec 8, 2023 at 8:37
  • Hi there, have you got a chance to check the answer below? May I know if it can address your concern? Commented Dec 8, 2023 at 13:25

1 Answer 1

1

Blob means files in this context.

I think your code can actually work if you set the -DestBlob.

EDIT: You can't copy a whole folder with the Start-AzStorageBlobCopy.


This is a sample code that retrieves all containers from storage account 1, and then creates the non-existing ones on storage account 2.

It then queries all files within the current container and copies them from SA1 to SA2.

Attention! the -Force option is included, so overwriting is possible. If you want to avoid this, just delete it.

#connect
Connect-AzAccount

#get all relevant information from the source
Set-AzContext -Subscription "SUB1"
$SourceResourceGroup = "SUB1_RG"
$SourceStorageAccount = "SUB1_SA"
$SourceStorageKey = (Get-AzStorageAccountKey -ResourceGroupName $SourceResourceGroup -Name $SourceStorageAccount)[0].Value
$SourceStorageContext = New-AZStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey

#get all relevant information from the destination
Set-AzContext -Subscription "SUB2"
$DestResourceGroup = "SUB2_RG"
$DestStorageAccount = "SUB2_SA"
$DestStorageKey = (Get-AzStorageAccountKey -ResourceGroupName $DestResourceGroup -Name $DestStorageAccount)[0].Value
$DestStorageContext = New-AZStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

#get all container from the source
$Containers = Get-AZStorageContainer -Context $SourceStorageContext

#do something with all container
foreach ($Container in $Containers) {
    $ContainerName = $Container.Name
    #Generate new container if does not exist
    if (!((Get-AZStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName })) {   
        Write-Output "Creating new container $ContainerName"
        New-AZStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }

    #get all file
    $Blobs = Get-AZStorageBlob -Context $SourceStorageContext -Container $ContainerName

    #Copy all file with overwrite! just delete the "-Force" if you don't want to overwrite your existing files
    foreach ($Blob in $Blobs) {
        $BlobName = $Blob.Name
        Write-Output "Copying $BlobName from $ContainerName"
        Start-AzStorageBlobCopy -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName -Force -ErrorAction SilentlyContinue
    }
}

If it was helpful please upvote it or accept the answer.

Sign up to request clarification or add additional context in comments.

3 Comments

the code doesn't quite do what I wanted to but I'm accepting for the comment "You can't copy a whole folder with the Start-AzStorageBlobCopy"
Thank you and sorry, I must have misunderstood something. What is your exact request? Isn't it to copy data from the DEV blob to the PROD blob? The script will copy everything, but with a little fine tuning you can make it more granular. Something like "if" or any filter etc.
Nothing wrong with what you've done @KKI it fits the scenario but it doesn't have the throughput that AZcopy does. I basically wanted to replicate the azcopy sync command in powershell as azcopy isn't as easy to handle errors in. I appreciate the help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.