Azure storage blob list with continuation token from specific point

Question

I'm working on a script to list the blobs in a container which has a ridiculous number of blobs (over 30 million!).

Anyway, I'm using the code from https://learn.microsoft.com/en-us/powershell/module/az.storage/get-azstorageblob?view=azps-3.8.0

Which appears to use a continuation token for every 10,000 files.

$MaxReturn = 10000
$ContainerName = "abc"
$Total = 0
$Token = $Null
do
 {
     $Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxReturn  -ContinuationToken $Token
     $Total += $Blobs.Count
     if($Blobs.Length -le 0) { Break;}
     $Token = $Blobs[$blobs.Count -1].ContinuationToken;
 }
 While ($Token -ne $Null)
Echo "Total $Total blobs in container $ContainerName"

The problem is that this always ends up hanging or getting stuck and never completes.

It usually gets around half way and I have to restart it which kicks off the entire process all over again.

However, I already have the data from the first run, is there a way to get it to start from a specific value rather than from the start?

Lets say I already have the records I need for the first 3 million blobs. How do I tell it to start from 3 million instead of 0?

Or am I not understanding how the process works?

2 Questions: 1) Are you storing the continuation token somewhere and 2) Is your data static i.e. blobs are not being added or deleted from your blob container? — Gaurav Mantri
– Gaurav Mantri, Commented May 12, 2020 at 5:59
I am not storing the continuation token, but I could. Are you suggesting I could use the last token (before it hangs) to continue? And the data in the container is static. — amaru96
– amaru96, Commented May 12, 2020 at 6:04
@amaru96a Since the data in the container is static, you can store the latest ContinuationToken. Then run the script with the ContinuationToken next time to get remaining blobs。 — Jim Xu
– Jim Xu, Commented May 12, 2020 at 7:51

Joey Cai · Accepted Answer · 2020-05-19 13:01:31Z

1

Just a summary for the issue to let others know who has the similar issue.

How do I tell it to start from 3 million instead of 0?

Since the data in the container is static, you can store the latest ContinuationToken. Then run the script with the ContinuationToken next time to get remaining blobs.

For more details you could refer to this article.

answered May 19, 2020 at 13:01

Joey Cai

20.2k1 gold badge25 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Azure storage blob list with continuation token from specific point

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related