How do configure azure table storage in local docker container?

Question

I am trying to set up a repository class that will create a table if it doesn't exist in my azure storage database. Here's the constructor:

    public TableStorageRepository(string storageAccountConnectionString, string tableName)
    {

        var options = new TableClientOptions
        {
            Retry =
            {
                MaxRetries = 5,
                Delay = TimeSpan.FromSeconds(2),
                MaxDelay = TimeSpan.FromSeconds(10),
                Mode = RetryMode.Exponential
            }
        };
       var serviceClient = new TableServiceClient(storageAccountConnectionString, options);
        _tableClient = serviceClient.GetTableClient(tableName);
        _tableClient.CreateIfNotExists();
    }

Here's the logic that calls it / set ups DI in Program.cs:

builder.Services.AddScoped<ITableStorageRepository<DeploymentDto>>(provider =>
{
    var connectionString = builder.Configuration["AzureTableStorage:ConnectionString"]; //use appsettings.[Development].json
    var tableName="Deployment";
    return new TableStorageRepository<DeploymentDto>(connectionString, tableName);
});

When I try to trigger a method on my controller the GETs deployments, the repository logic is trigger but it fails when it tries the CreateifNotExists() method. The error is:

System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An error occurred while sending the request.) (An error occurred while sending the request.) (An error occurred while sending the request.) (An error occurred while sending the request.) Azure.RequestFailedException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) End of inner exception stack trace --- at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) End of inner exception stack trace ---

Any suggestions would be appreciated.

I tried to change the code so instead of a connection string, it passes individual parameters. Soething like this:

    public TableStorageRepository(string accountName, string accountKey, string tableName, string endpointUri = null)
    {
        var serviceUri = endpointUri != null ? new Uri(endpointUri) : new Uri($"https://{accountName}.table.core.windows.net");
        var credential = new TableSharedKeyCredential(accountName, accountKey);
        _tableClient = new TableClient(serviceUri, tableName, credential);
                // Ensure the table exists asynchronously
        _tableClient.CreateIfNotExistsAsync().GetAwaiter().GetResult();
    }

I get the same results - which makes sense because I think the connection itself is being made correctly. At least, thats my interpretation .

The next thing i started to check out was to see if there's a way to try to use the api to call database methods directly, just to see if the storage system responds.
I have the following section in docker-compose.yml:

azureTableStorage:
    image: mcr.microsoft.com/azure-storage/azurite:latest #Todo: replace with a real version once everyting is working
    container_name: azureTableStorage
    ports:
      - "10000:10000" # Blob service
      - "10001:10001" # Queue service
      - "10002:10002" # Table service      
    command: "azurite-table --loose --location /data --debug /dev/stdout"
    volumes:
      - .azureTableStorage/data:/azureTableStorage/data
    environment:
      - AZURITE_ACCOUNTS=devstoreaccount1:Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6 #sdk expects base64.  But docker file just expects plain

The container starts up no problems. But ... I'm not able to run any curl commands inside the container (or outside of it) to see the table data.

Here's what I tried inside the container:

</Error>/opt/azurite # curl -X GET \
>   -H "Authorization: SharedKey devstoreaccount1:RWJ5OHZkTTAyeE5PY3FGZXZrYlBaUnZ6MUZjbEo3dDNZdnNaNkVieTh2ZE0wMnhOT2NxRmV2a2JQWlJ2ejFGY2xKN3QzWXZzWjY=" \
>   -H "Accept: application/json;odata=fullmetadata" \
>   -H "x-ms-version: 2019-02-02" \
>   http://127.0.0.1:10002/devstoreaccount1/Tables
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>AuthorizationFailure</Code>
  <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.
RequestId:5376245b-4fd1-4db9-992f-6bb88c1c658c
Time:2025-02-10T19:22:24.822Z</Message>
</Error>/opt/azurite #

The key value i'm sending is the value i get back from this powershell command:
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6"))

EDIT 1

I have followed the instructions.

Updated my docker-compose to include the environment variable.
My Program.cs looks like this:
The correct connnection string is being passed from the appsettings.Development.json:

But the constructor fails with the error:

Exception has occurred: CLR/System.FormatException An exception of type 'System.FormatException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.'
at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength) at System.Convert.FromBase64String(String s) at Azure.Data.Tables.TableSharedKeyCredential.SetAccountKey(String accountKey) at Azure.Data.Tables.TableSharedKeyCredential..ctor(String accountName, String accountKey) at Azure.Data.Tables.TableConnectionString.GetCredentials(ConnectionString settings) at Azure.Data.Tables.TableConnectionString.ParseInternal(String connectionString, TableConnectionString& accountInformation, String& error) at Azure.Data.Tables.TableConnectionString.Parse(String connectionString) at Azure.Data.Tables.TableClient..ctor(String connectionString, String tableName, TableClientOptions options) at Azure.Data.Tables.TableClient..ctor(String connectionString, String tableName) at Provider.Gateway.Infrastructure.Storage.TableStorage.TableStorageRepository`1..ctor(String connectionString, String tableName) in /Users/me/src/projects/provider-gateway/Provider.Gateway/Infrastructure/Storage/TableStorageRepository.cs:line 46

EDIT 2

I've updated my appsettings.Development.json file to look like this:

 {
     "AzureTableStorage": {
         "ConnectionString": "UseDevelopmentStorage=true"
     }
 }

And now when I run i don't get any errors. I can POST and GET. But the last thing I need to document for the team is a way to "see" the data in the container, outside the context of our web application. So I downloaded azure storage explorer.

And I tried to create a connection to a local storage container. But i think it's connecting to my development box (host) instance of azurite and not the one running in my container.

I've proven this to myself by debugging my app, GETting the list of deployments. Then I do a docker compose down on all my containers and I redo the GET and it still works. I opened the azure extension in vscode and I can see that it was creating it locally on my dev box ("host" machine to the container).

In case it helps, when I do a docker ps I get the following:

CONTAINER ID   IMAGE                                            COMMAND                  CREATED         STATUS                   PORTS                                       NAMES
c13cdb5f263b   ghcr.io/open-webui/open-webui:main               "bash start.sh"          2 minutes ago   Up 2 minutes (healthy)   0.0.0.0:3000->8080/tcp                      apg-open-webui-1
a9d70b8d1600   ollama/ollama:0.5.7                              "/bin/ollama serve"      2 minutes ago   Up 2 minutes             0.0.0.0:11434->11434/tcp                    apg-ollama-1
a61983d56ed4   mcr.microsoft.com/azure-storage/azurite:latest   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes             10000-10001/tcp, 0.0.0.0:10002->10002/tcp   azureTableStorage

The other artifact I found that proves its not writing to the container is that i have a azurite_db_table.json file in the root of the project folder instead of inside the mapped volume defined in the docker file for this container.

EDIT 3

Your suggestion for the volumes tag in the docker-compose is giving me errors - likely I'm not using the correct syntax:

The first step is to ensure that you can connect to Azurite. You should be able to access the local queue using Azure Storage Explorer: learn.microsoft.com/en-us/azure/storage/blobs/… If you are unable to connect, try removing the Azurite command and environment settings from your Docker Compose setup. The connection string is set by default: learn.microsoft.com/en-us/azure/storage/common/… — Florian Vuillemot
– Florian Vuillemot, Commented Feb 11 at 6:40
Can you please check "ConnectionString":"UseDevelopmentStorage=true;"in application json and check — P Sampath
– P Sampath, Commented Feb 11 at 17:42
@Sampath I guess I don't understand what you mean when you say "I have change to "ConnectionString":"UseDevelopmentStorage=true;"" Can you update your answer to clarify? — J Lee
– J Lee, Commented Feb 12 at 21:56
Make sure to Run the Azurite Docker image using this doc you can connectionstring like this as given in the git — P Sampath
– P Sampath, Commented Feb 13 at 3:17

P Sampath · Accepted Answer · 2025-02-14 16:04:47Z

1

Below are the steps to configuring Azure Table Storage in a local Docker container using Azurite with C# .

Setup Azurite in Docker by adding docker-compose.yml**

Create a docker-compose.yml file to define the Azurite container.

version: "3.8"

services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite:latest
    container_name: azurite
    ports:
      - "10000:10000"  # Blob service
      - "10001:10001"  # Queue service
      - "10002:10002"  # Table service
    command: "azurite-table --loose --location /data --debug /dev/stdout"
    volumes:
      - ./data:/data
    environment:
      - AZURITE_ACCOUNTS=devstoreaccount1:Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6

Run the container:

docker-compose up -d

Set this in appsettings.Development.json:

{ "AzureTableStorage": { "ConnectionString": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;" } }

I have change to "ConnectionString":"UseDevelopmentStorage=true;"

Install Required NuGet Packages below

dotnet add package Azure.Data.Tables

Create an interface for your repository:

public interface ITableStorageRepository<T> where T : class, ITableEntity, new()
{
    Task AddEntityAsync(T entity);
    Task<List<T>> GetAllEntitiesAsync();
}

Create a repository to interact with Azure Table Storage:

using Azure.Data.Tables;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

public class TableStorageRepository<T> : ITableStorageRepository<T> where T : class, ITableEntity, new()
{
    private readonly TableClient _tableClient;

    public TableStorageRepository(string connectionString, string tableName)
    {
        _tableClient = new TableClient(connectionString, tableName);
        _tableClient.CreateIfNotExists();
    }

    public async Task AddEntityAsync(T entity)
    {
        await _tableClient.AddEntityAsync(entity);
    }

    public async Task<List<T>> GetAllEntitiesAsync()
    {
        return _tableClient.Query<T>().ToList();
    }
}

add Dependency Injection in Program.cs**

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Configuration;

var builder = WebApplication.CreateBuilder(args);

var connectionString = builder.Configuration["AzureTableStorage:ConnectionString"];
builder.Services.AddScoped(typeof(ITableStorageRepository<>), provider =>
    new TableStorageRepository<dynamic>(connectionString, "Deployment"));

var app = builder.Build();
app.Run();`

Create an API controller to expose endpoints for interacting with Azure Table Storage.

using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using System.Threading.Tasks;

[Route("api/deployments")]
[ApiController]
public class DeploymentController : ControllerBase
{
    private readonly ITableStorageRepository<DeploymentEntity> _repository;

    public DeploymentController(ITableStorageRepository<DeploymentEntity> repository)
    {
        _repository = repository;
    }

    [HttpPost]
    public async Task<IActionResult> AddDeployment([FromBody] DeploymentEntity entity)
    {
        await _repository.AddEntityAsync(entity);
        return Ok();
    }

    [HttpGet]
    public async Task<IActionResult> GetDeployments()
    {
        var result = await _repository.GetAllEntitiesAsync();
        return Ok(result);
    }
}

Define an Entity Model like below

using Azure;
using Azure.Data.Tables;

public class DeploymentEntity : ITableEntity
{
    public string PartitionKey { get; set; } = "Deployment";
    public string RowKey { get; set; } = Guid.NewGuid().ToString();
    public string Name { get; set; }
    public string Status { get; set; }

    public DateTimeOffset? Timestamp { get; set; }
    public ETag ETag { get; set; }
}

Test with CURL or Postman:

curl -X POST "http://localhost:5000/api/deployments" -H "Content-Type: application/json" -d '{ "Name": "Deployment1", "Status": "Completed" }'

Docker

Update 1: for The error "The input is not a valid Base-64 string Use ppsettings.Development.json file to look like this:

 {
     "AzureTableStorage": {
         "ConnectionString": "UseDevelopmentStorage=true"
     }
 }

Update 2: To get data in exec of docker change the docker-compose.yml like sample example below:

volumes: - azure-storage-volume:/data and command: azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 --location /data --debug /data/debug.log

Then Stop and remove existing containers by docker-compose down -v

Rebuild and restart the container docker-compose up -d

Then use docker exec -it azure-blob-storage sh -c "ls -lah /data"

Ouput
and use docker exec -it azure-blob-storage sh -c "cat /data/__azurite_db_table__.json"

Output

Docker

edited Feb 14 at 16:04

answered Feb 11 at 13:16

P Sampath

4,2832 gold badges7 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

J Lee Feb 11 at 15:59

Please see EDIT 1 in my post

J Lee Feb 13 at 14:18

Just one other related question. Instead of the curl call, can you suggest something inside the docker container where I can see that the records I'm adding are being created? I just need a smoking gun to prove that the records are being created inside the docker container and not on some local instance of azure tables running on the host.

P Sampath Feb 13 at 14:53

instead of using a curl call, you can use Azure Storage Explorer to verify that the records are being created inside the Docker container. Azure Storage Explorer is a free tool. you see the data as show in the image while docker image /container running.

J Lee Feb 13 at 15:18

Ive just proven to myself that the data is not being saved in the container. But it's using the azurite instance running on the dev box (host machine to container. Please see edit 2).

P Sampath Feb 14 at 15:45

to get in exc of docker change the like sample below ` volumes: - azure-storage-volume:/data` and command: azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 --location /data --debug /data/debug.log

|

J Lee · Accepted Answer · 2025-02-17 14:40:43Z

I think the root cause of the issue was that I had azurite running locally on my "host machine". but I also had it containerized. When I start debugging in vscode, it would automatically start the local instance on the host machine because in appsettings.Development.json I had the following:

{
    "AzureTableStorage": {
        "ConnectionString": "UseDevelopmentStorage=true"
    }
}

I replaced that with:

{
    "AzureTableStorage": {
      "ConnectionString": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;TableEndpoint=http://localhost:10002/devstoreaccount1"
    }
  }

Now when I start vscode, it doesn't automatically start the azure table service on my host. I can tell because on the status bar at the bottom of vscode, it doesn't show the service running.

And now when i create a new "deployment" record in the database using swagger, I can prove that it's updating the container data using the tip from the other answer:

/opt/azurite # cat /data/__azurite_db_table__.json | grep Deployment
{"filename":"/data/__azurite_db_table__.json","collections":[{"name":"$TABLES_COLLECTION$","data":[{"account":"devstoreaccount1","table":"Deployment","meta":{"revision":0,"created":1739802123425,"version":0},"$loki":1}],"idIndex":null,"binaryIndices":{"account":{"name":"account","dirty":false,"values":[0]},"table":{"name":"table","dirty":false,"values":[0]}},"constraints":null,"uniqueNames":[],"transforms":{},"objType":"$TABLES_COLLECTION$","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":1,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]},{"name":"$SERVICES_COLLECTION$","data":[],"idIndex":null,"binaryIndices":{},"constraints":null,"uniqueNames":["accountName"],"transforms":{},"objType":"$SERVICES_COLLECTION$","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":0,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]},{"name":"devstoreaccount1$Deployment","data":[{"PartitionKey":"test","RowKey":"test","properties":{"PartitionKey":"test","RowKey":"test","Id":"test","Provider":"AppsAIGateway","Endpoint":"https://mytest.com","Version":"string","Timestamp":"2025-02-17T14:36:11.9931129Z","[email protected]":"Edm.DateTime"},"lastModifiedTime":"2025-02-17T14:36:11.9931129Z","eTag":"W/\"datetime'2025-02-17T14%3A36%3A11.9931129Z'\"","meta":{"revision":0,"created":1739802971998,"version":0},"$loki":1}],"idIndex":null,"binaryIndices":{"PartitionKey":{"name":"PartitionKey","dirty":false,"values":[0]},"RowKey":{"name":"RowKey","dirty":false,"values":[0]}},"constraints":null,"uniqueNames":[],"transforms":{},"objType":"devstoreaccount1$Deployment","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":1,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]}],"databaseVersion":1.5,"engineVersion":1.5,"autosave":true,"autosaveInterval":5000,"autosaveHandle":null,"throttledSaves":true,"options":{"persistenceMethod":"fs","autosave":true,"autosaveInterval":5000,"serializationMethod":"normal","destructureDelimiter":"$<\n"},"persistenceMethod":"fs","persistenceAdapter":null,"verbose":false,"events":{"init":[null],"loaded":[],"flushChanges":[],"close":[],"changes":[],"warning":[]},"ENV":"NODEJS"}
/opt/azurite #

Collectives™ on Stack Overflow

How do configure azure table storage in local docker container?

2 Answers 2

10 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related