How to resume data copy from blob using a stored continuation token?

Question

I have been trying to copy data from blobs in multiple batches referring the below code in powershell.

    $MaxReturn = 10000
    $ContainerName = "abc"
    $Total = 0
    $Token=New-Object -TypeName 'Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken'
    $Token.NextMarker='last stored token value'
    do
     {
         $Blobs = Get-AzureStorageBlob -Container $ContainerName -MaxCount $MaxReturn  -ContinuationToken $Token
         $Total += $Blobs.Count
         if($Blobs.Length -le 0) { Break;}
         $Token = $Blobs[$blobs.Count -1].ContinuationToken;
     }
     While ($Token -ne $Null)
    Echo "Total $Total blobs in container $ContainerName"

I am saving the value of $Token after each batch in a file.I want to restart copying data from the continuation token which was last saved in the file.

To achieve this, i am manually replacing the line $Token=null with $Token="last saved token" and it throws me an error

"Cannot bind parameter 'ContinuationToken'. Cannot convert the value of type "System.String" to type "Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken".

How can i pass the value of last saved token to ContinuationToken parameter of Get-AzureStorageBlob ?

I have tried using

$Token=New-Object -TypeName Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken -NextMarker "Token value"

but it again throws an error

Cannot find type [Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken]: verify that the assembly containing this type is loaded.

Below is the powershell command to check for the same.

PS C:\Users\F\Documents> [Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken]

How are you serializing the ContinuationToken? Can you edit your question and include how the serialized token looks like? — Gaurav Mantri
– Gaurav Mantri, Commented Mar 3, 2020 at 13:07
Hi Gaurav, thanks for your response. I am not able to serialize the continuation token. I have updated the question with how i tried. Initially i was copying only the string value and now i am trying to create an object of type [Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken] and pass that string value into the NextMarker property to see if it solves the problem. — Faizan
– Faizan, Commented Mar 3, 2020 at 14:19
It seems that the class[Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken] was not loading before Get-AzureStorageBlob was called. Therefore it was giving me this error. I added this line "Import-Module -Name AzureRM" in my script at the top and the code picks up the continuation token and resumes the copy process from where it ended. Thanks for your hint Gaurav. — Faizan
– Faizan, Commented Mar 3, 2020 at 15:27

Faizan · Accepted Answer · 2020-03-03 15:30:29Z

1

Import-Module -Name AzureRM
$MaxReturn = 10000
    $ContainerName = "abc"
    $Total = 0
    $Token=New-Object -TypeName 'Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken'
    $Token.NextMarker='last stored token value'
    do
     {
         $Blobs = Get-AzureStorageBlob -Container $ContainerName -MaxCount $MaxReturn  -ContinuationToken $Token
         $Total += $Blobs.Count
         if($Blobs.Length -le 0) { Break;}
         $Token = $Blobs[$blobs.Count -1].ContinuationToken;
     }
     While ($Token -ne $Null)
    Echo "Total $Total blobs in container $ContainerName"

I needed to serialize the continuation token and import the AzureRM module to get the copy process start from the last value of continuation token. Thanks for all the help.

answered Mar 3, 2020 at 15:30

Faizan

10515 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Anoop R Desai Over a year ago

For what it's worth, AzureRM is deprecated going forward.

Vad · Accepted Answer · 2020-03-03 13:30:16Z

0

This parameter is used for batches as i can find in documentation. So you can't convert it from any type to [Microsoft.WindowsAzure.Storage.Blob.BlobContinuationToken] May be you find some help in link below, but i find that object have methods to write and read from xml file,so it's chanse to convert a serializable continuation token into its XML representation and then read it. link1

answered Mar 3, 2020 at 13:30

Vad

7434 silver badges13 bronze badges

1 Comment

Faizan Over a year ago

Hi Vad. I am able to serialize the continuation token and get the code working. Thanks for your help.

Collectives™ on Stack Overflow

How to resume data copy from blob using a stored continuation token?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related