Connection exception in asynchronous processing using Dapper, Channels, and BackgroundServices in .NET 6

I am working on an ASP.NET Core 6 Web API that uses Dapper and performs long-running processes dependent on balance checks to decide whether they can proceed or not. Previously, we identified an issue where simultaneous executions could result in a negative total balance:

1. The first execution consumed the entire available balance.
2. The second execution had already validated the balance before the first finished, allowing it to proceed despite insufficient balance.

To address this, we implemented a solution using a BackgroundService and Channel, creating keyed queues to ensure the calls are processed sequentially. Due to cost restrictions and technical decisions, we couldn't use external queue solutions like RabbitMQ or Azure Service Bus.

Current situation

The solution works well in most cases, especially in our staging environment. However, in production, we occasionally see the following exception thrown by the first async method:

Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.

This issue occurs only in production, where the server is more robust (higher processing power). Our initial investigation compared the environment configurations, but the only differences identified were related to infrastructure (a more powerful VPS in production).

The processing structure is as follows:

• The controller that receives the request enqueues the task in the Channel and waits for the queue to complete.

private readonly IQueuedHostedServiceProcessor _processor;

[HttpPost("{id}/some-task")]
public async Task<IActionResult> SomeTaskAsync([FromRoute] long id, [FromBody] SomeDTO someDto)
{
    var key = "FunctionKey";

    async Task<bool> fnProcess()
    {
        using var scope = _scopeFactory.CreateAsyncScope();
        
        var requiredService = scope.ServiceProvider.GetService<IRequiredService>();
        
        return await requiredService.Process(id, someDto);
    }

    return await _processor.InvokeAsync(key, fnProcess);
}

• To avoid dependency lifecycle issues, we create a new DI scope for each execution.
• The processing method is asynchronous, returning a Task<T>, but it mixes synchronous and asynchronous calls to the database.

//RequiredService
public async Task<bool> Process(long id, SomeDTO someDto){
   var a = _repositoryA.GetById(id);
   var b = await _repositoryA.CheckSomething(someDto); <- throws exception
   var c = await _serviceC.DoSomething(someDto);

   return c;
}

The IQueuedHostedServiceProcessor implementation is based on this code and this article

Hypothesis

My hypothesis is that mixing synchronous and asynchronous calls is causing resource management issues, particularly with database connections. This could explain the intermittent nature of the issue, as .NET might be struggling to manage the connection lifecycle.

Interestingly, if I retry the same operation immediately after the failure, the issue doesn’t occur. This leads me to suspect that it might be related to how connections are released or reused, possibly involving the garbage collector.

What we're doing

We are gradually migrating all methods and dependencies to be fully asynchronous, but this is a long process due to the large codebase and limited team resources.

We investigated differences between the environments, but the only significant variable is the superior performance in production.

Questions

1. Is my hypothesis about mixing synchronous and asynchronous calls correct? If so, how does it impact the lifecycle of connections?

2. Are there alternatives to resolve or mitigate this issue without relying solely on migrating to fully asynchronous methods?

3. Could this behavior be related to Dapper or the Channel, especially in environments with higher performance?

4. Does the fact that the error doesn’t occur when retrying immediately indicate an issue with the garbage collector or other resource management aspects?

Any guidance on how to better understand the issue or optimize this solution would be greatly appreciated. Thank you!