List files on Azure BlobStorage with python API

Question

This code tries to list the files in the in a blob storage:

#!/usr/bin/env python3

import os

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
from datetime import datetime, timedelta
import azure.cli.core as az

print(f"Azure Blob storage v{__version__} - Python quickstart sample")

account_name = "my_account"
container_name = "my_container"
path_on_datastore = "test/path"

def _create_sas(expire=timedelta(seconds=10)) -> str:
    cli = az.get_default_cli()

    expire_date = datetime.utcnow() + expire
    expiry_string = datetime.strftime(expire_date, "%Y-%m-%dT%H:%M:%SZ")
    cmd = ["storage", "container", "generate-sas", "--name", container_name, "--account-name",
           account_name, "--permissions", "lr", "--expiry", expiry_string, "--auth-mode", "login", "--as-user"]
    if cli.invoke(cmd) != 0:
        raise RuntimeError("Could not receive a SAS token for user {}@{}".format(
            account_name, container_name))

    return cli.result.result


sas = _create_sas()
blob_service_client = BlobServiceClient(
    account_url=f"{account_name}.blob.core.windows.net", container_name=container_name, credential=sas)

container_client = blob_service_client.create_container(container_name)
blob_list = container_client.list_blobs()
for blob in blob_list:
    print("\t" + blob.name)

That code worked quite fine a few weeks ago, but then we always get the error:

azure.core.exceptions.ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Does someone know what can be wrong?

PS. using Azure blob storage package of version 12.3.2.

[Edit]

Because of security concerns we are not allowed to use account keys here.

RoadRunner · Accepted Answer · 2020-11-04 14:08:29Z

7

I'm not entirely sure what is wrong with your code, but it looks like your SAS token is not the expected format. Have you tested if the the SAS URL works in a browser?

Additionally, your _create_sas function seems to be creating the SAS signature with an Azure CLI command. I don't think you need to do this because the azure-storage-blob package has methods such as generate_account_sas to generate a SAS signature. This will eliminate a lot of complexity because you don't need to worry about the SAS signature format.

from datetime import datetime, timedelta
from azure.storage.blob import (
    BlobServiceClient,
    generate_account_sas,
    ResourceTypes,
    AccountSasPermissions,
)
from azure.core.exceptions import ResourceExistsError

account_name = "<account name>"
account_url = f"https://{account_name}.blob.core.windows.net"
container_name = "<container name>"

# Create SAS token credential
sas_token = generate_account_sas(
    account_name=account_name,
    account_key="<account key>",
    resource_types=ResourceTypes(container=True),
    permission=AccountSasPermissions(read=True, write=True, list=True),
    expiry=datetime.utcnow() + timedelta(hours=1),
)

Which gives this SAS signature read, write and list permissions on blob containers, with an expiry time of 1 hour. You can change this to your liking.

We can then create the BlobServiceClient with this SAS signature as a credential, then create the container client to list the blobs.

# Create Blob service client to interact with storage account
# Use SAS token as credential
blob_service_client = BlobServiceClient(account_url=account_url, credential=sas_token)

# First try to create container
try:
    container_client = blob_service_client.create_container(name=container_name)

# If container already exists, fetch the client
except ResourceExistsError:
    container_client = blob_service_client.get_container_client(container=container_name)

# List blobs in container
for blob in container_client.list_blobs():
    print(blob.name)

Note: The above is using version azure-storage-blob==12.5.0, which is the latest package. This is not too far ahead of your version, so I would probably update your code to work with latest functionality, as also provided in the documentation.

Update

If you are unable to use account keys for security reasons, then you can create a service principal and give it a Storage Blob Data Contributor role to your storage account. This gets created as an AAD application, which will have access to your storage account.

To get this setup, you can use this guide from the documentation.

Sample Code

from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient
token_credential = DefaultAzureCredential()

blob_service_client = BlobServiceClient(
    account_url="https://<my_account_name>.blob.core.windows.net",
    credential=token_credential
)

edited Nov 4, 2020 at 14:08

answered Nov 3, 2020 at 16:08

RoadRunner

26.4k6 gold badges46 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Matthias Over a year ago

Thank You for your fast response. Is there a possibility to create the sas token without an account key?

RoadRunner Over a year ago

@Matthias I don't believe so. If you end goal is to just list blobs, then you can probably use a service principal to authenticate with Azure AD. This goes through that method: github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/….

Matthias Over a year ago

The end goal is to list and provide a download link (with sas) of the files. Because of security concerns we are not allowed to use the account key.

RoadRunner Over a year ago

@Matthias Understood. I have provided an update to show you how to use a service principal. If you want the most secure option, then managed identities is best here, since it uses no credentials.

fcdt · Accepted Answer · 2020-11-04 09:13:52Z

0

It looks like the module is deprecated:

Starting with v5.0.0, the 'azure' meta-package is deprecated and cannot be installed anymore. Please install the service specific packages prefixed by azure needed for your application.

The complete list of available packages can be found at: https://aka.ms/azsdk/python/all

A more comprehensive discussion of the rationale for this decision can be found in the following issue: https://github.com/Azure/azure-sdk-for-python/issues/10646

edited Nov 4, 2020 at 9:13

fcdt

2,5215 gold badges17 silver badges27 bronze badges

answered Nov 3, 2020 at 15:44

Judev1

4343 silver badges11 bronze badges

Collectives™ on Stack Overflow

List files on Azure BlobStorage with python API

[Edit]

2 Answers 2

Update

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

[Edit]

2 Answers 2

Update

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related