Enumerating large Azure BLOB containers using Azure.Storage.Blobs

Question

With Microsoft.WindowsAzure.Storage.Blob one could enumerate a large BLOB container using the BlobContinuationToken in the ListBlobsSegmentedAsync method, and the BlobResultSegment would include the ContinuationToken property if a subsequent read would be required to retrieve more from the container.

With Azure.Storage.Blobs it appears the enumeration API is BlobContainerClient.GetBlobsAsync but the method signature does not provide a means of supplying a continuation token, and the results as Azure.AsyncPageable<BlobItem> do not appear to provide a property that is a continuation token.

Does anyone have a code snippet for enumerating large BLOB containers on Azure, using continuation tokens, via the Azure.Storage.Blobs package?

Peter Bons · Accepted Answer · 2023-01-05 07:32:09Z

3

If you really want to use a continuation token you can do so:

    string continuationToken = null;
    var container = new BlobContainerClient("xxx", "test");
    var blobPages = container.GetBlobsAsync().AsPages();
        
    await foreach (var page in blobPages)
    {
        continuationToken = page.ContinuationToken;
        foreach(var blob in page.Values)
        {
            ...
        }
    }

You can pass the token to AsPages to continue iteration:

blobPages = container.GetBlobsAsync().AsPages(continuationToken: continuationToken);

or you can just iterate over the blobs, let the SDK do the job of providing all the data using a stream of blobs:

    var container = new BlobContainerClient("xxx", "test");
    var blobs = container.GetBlobsAsync();
        
    await foreach (var blob in blobs)
    {
        //
    }

Iterating over the blobs can make multiple calls to the blob service.

For more about pagination in the SDK see the docs.

edited Jan 5, 2023 at 7:32

answered Jan 5, 2023 at 7:26

Peter Bons

30k4 gold badges64 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

joelc Over a year ago

Thank you, exactly what I needed. Yes, I agree the best approach would be to allow the SDK to do its job, but in this case, the implementation expects a continuation token. Thanks for putting this answer together Peter!

Collectives™ on Stack Overflow

Enumerating large Azure BLOB containers using Azure.Storage.Blobs

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related