Unable to delete Multiple Blobs from Container in Python

Question

I am trying to delete blobs in my containers. Each container has minimum 1500-2000 blobs. Each container contains jpeg files and one mp4 file. If the mp4 file is present, only then I will delete the blobs inside that particular container. Every time when I try to execute content.delete_blobs(*blobsToDelete) ,I get the following exception:

Exception in Non AI : Traceback (most recent call last):
  File "deleteblobs.py", line 278, in BlobsToDeleteeNonAI
    content.delete_blobs(*blobsToDelete)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\storage\blob\_container_client.py", line 1194, in delete_blobs
    return self._batch_send(*reqs, **options)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 304, in _batch_send
    raise error
azure.storage.blob._shared.response_handlers.PartialBatchErrorException: There is a partial failure in the batch operation.

Here is what my code looks like :

def BlobsToDeleteeNonAI():
        blobsToDelete = []
        #Will delete all the photos except the video.
        try:
                for containerName in NonAICandidates:
                        try:
                                mp4Found = 0
                                content = blob_service_client.get_container_client(str(containerName))
                                for blobs in content.list_blobs():
                                        print("\n"+blobs.name)
                                        #file.write("\n" +blobs.name)
                                        if(blobs.name.endswith(".jpeg")):   #str(blobs.size)
                                                blobsToDelete.append(blobs.name)

                                        if(blobs.name.endswith(".mp4")):
                                                mp4Found = 1
                                                file.write("\nMP4 File Name : " +str(blobs.name))
                                #Will only Delete if and only if the Video File is Present
                                if(mp4Found == 1):
                                        #DeleteCodeHere
                                        
                                        file.write("\n Mp4 Found : " +str(mp4Found) + " for " +str(containerName))
                                        #file.write("\n Blobs to Delete : "+str(blobsToDelete))
                                        
                                        content.delete_blobs(*blobsToDelete)
                                        blobsToDelete.clear()
                                        file.write("\n Blobs Deleted for : " +str(containerName))
                                else:
                                        file.write("\nMp4 File Not found for Non AI Candidate : " +str(containerName) + ". Cannot Perform Deletion Operation.");
                                                
                                           
                        except:
                                file.write("\nException in Non AI : " +str(traceback.format_exc()))
                                blobsToDelete.clear()
        except:
                 file.write("\nException : " +str(traceback.format_exc()))




if __name__ == "__main__":

        NonAICandidates = ['container1', 'container2', 'container3', 'container4', 'container5', 'container6', ....]


        BlobsToDeleteeNonAI()

Is there anything wrong with the implementation or is there any other issue which is preventing me from deleting the blobs ?

Ivan Glasenberg · Accepted Answer · 2020-11-09 08:16:44Z

4

The reason is that a single batch can only support 256 sub-requests. And in your container, there are at least 1500-2000 blobs to be deleted, when you try to delete these blob in one delete_blobs method, it's beyond the 256 limitations.

You should modify your code, in one delete_blobs method, only delete 1 to 256 blobs. Here is an sample:

#Will only Delete if and only if the Video File is Present
if(mp4Found == 1):
    blobs_lenth=len(blobsToDelete)

    start=0
    end=256

    while end<=blobs_lenth:
         #each time, delete 256 blobs at most
         container_client.delete_blobs(*blobsToDelete[start:end])
         start = start + 256
         end = end + 256

         if start < blobs_lenth and end > blobs_lenth:
            container_client.delete_blobs(*blobsToDelete[start:blobs_lenth])

answered Nov 9, 2020 at 8:16

Ivan Glasenberg

30.3k2 gold badges56 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

S M Over a year ago

Thanks for the help. After making relevant changes, I am able to delete blobs from a container who have more than 256 blobs. Thanks once again.

NoUserException Over a year ago

FYI - If using the code above, there is a bug that will cause it to not delete if length < 256.

Viv Over a year ago

how to avoid exception in cases we have some blob name in a list but in actuality it does not exist in container.. we get There is a partial failure in the batch operation.

Alex Metsai · Accepted Answer · 2021-06-16 19:34:59Z

1

if(mp4Found == 1):
    blobs_lenth=len(blobsToDelete)

    if blobs_lenth <= 256:
        container_client.delete_blobs(*blobsToDelete) 

    else:
        start=0
        end=256

        while end<=blobs_lenth:
             #each time, delete 256 blobs at most
             container_client.delete_blobs(*blobsToDelete[start:end])
             start = start + 256
             end = end + 256

             if start < blobs_lenth and end > blobs_lenth:
                container_client.delete_blobs(*blobsToDelete[start:blobs_lenth])

edited Jun 16, 2021 at 19:34

Alex Metsai

1,9605 gold badges15 silver badges26 bronze badges

answered Jun 16, 2021 at 13:42

Saurabh Kumar

111 bronze badge

1 Comment

Saurabh Kumar Over a year ago

This will handle cases even if length of blob is less then 256

HORIUCHI Satoshi · Accepted Answer · 2021-11-07 02:00:57Z

1

I found this sample can be simply written like as below. works fine.

if(mp4Found == 1):
    blobs_length=len(blobsToDelete)

    for i in range(0, blobs_length, 256):
        container_client.delete_blobs(*blobsToDelete[i: i+256])

answered Nov 7, 2021 at 2:00

HORIUCHI Satoshi

111 bronze badge

Collectives™ on Stack Overflow

Unable to delete Multiple Blobs from Container in Python

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related