Python Azure SDK: Using list_blobs to get more than 5.000 results

Question

I'm having trouble with the Python Azure SDK and haven't found anything, neither on Stack Overflow nor in the MSDN Forums.

I want to use Azure SDKs list_blobs() to get a list of blobs. There are more than 5,000 (which is the max_result).

If I take a look at the code in the SDK itself, then I see the following:

def list_blobs(self, container_name, prefix=None, marker=None,
               maxresults=None, include=None, delimiter=None):

The description for 'Marker' being:

marker:
    Optional. A string value that identifies the portion of
    the list to be returned with the next list operation.
    The operation returns a marker value within the response
    body if the list returned was not complete. The marker
    value may then be used in a subsequent call to request
    the next set of list items. The marker value is opaque
    to the client.

My problem is that I'm unaware on how to use the marker to get the next set of 5,000 results. If I try something like this:

blobs = blobservice.list_blobs(target_container, prefix= prefix)
print(blobs.marker)

Then the marker is always empty, which I assume is because list_blobs() already parses the blobs out of the response.

But if that is the case then, how do I actually use the marker in a meaningful way?

Gaurav Mantri · Accepted Answer · 2014-06-19 09:51:33Z

5

SDK returns the continuation token in a variable called next_marker. You should use that to get the next set of blobs. See the code below as an example. Here I'm listing 100 blobs from a container at a time:

from azure import *
from azure.storage import *

blob_service = BlobService(account_name='<accountname>', account_key='<accountkey>')
next_marker = None
while True:
    blobs = blob_service.list_blobs('<containername>', maxresults=100, marker=next_marker)
    next_marker = blobs.next_marker
    print(next_marker)
    print(len(blobs))
    if next_marker is None:
        break
print "done"

P.S. The code above throws an exception on the last iteration. Not sure why. But it should give you an idea.

answered Jun 19, 2014 at 9:51

Gaurav Mantri

137k17 gold badges247 silver badges282 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user3755680 Over a year ago

Oh, it is 'next marker'! I actually feel pretty stupid now. How did you figure that out? Just so that I can make sure to solve my next problem on my own :)

Gaurav Mantri Over a year ago

Don't feel bad :) ... I ended up modifying the code here github.com/Azure/azure-sdk-for-python/blob/master/azure/storage/… to see the properties returned. Not very obvious I must say. For future reference, you can look at the source code of the SDK here: github.com/Azure/azure-sdk-for-python. HTH.

user3755680 Over a year ago

Ah, I see! I actually checked the source code for the list_blobs - function but I never had the idea of checking the parsing code >.< Anyway - thank you for the help!

Piotr Over a year ago

In order to get rid of this error you need to change line if next_marker is None: to if not next_marker:.

Ashish Anand · Accepted Answer · 2021-09-13 19:33:14Z

0

from azure import *
from azure.storage import *

block_blob_service = BlockBlobService(account_name=AccountName, account_key=AccountKey)
names= []
next_marker = None
while True:
    blob_list = block_blob_service.list_blobs(container,  prefix='/curated/projects/IntelligentShipmentOptimization/',
                                    num_results=400, marker=next_marker)
    for blob in blob_list:
        names.append(blob.name)
    
    next_marker = blob_list.next_marker
    print(next_marker)
    if not next_marker:
        break

answered Sep 13, 2021 at 19:33

Ashish Anand

2,76326 silver badges15 bronze badges

Collectives™ on Stack Overflow

Python Azure SDK: Using list_blobs to get more than 5.000 results

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related