ConcurrentQueue dequeue problem in Azure function

Question

I have declared a ConcurrentQueue and added a list of GUIDs. Adding into the queue is fine, but when I access the queue from inside the TimerTrigger function it seems like it's empty (updateQueue.count is 0). This behavior is happening in the cloud, but when I execute the same locally it works fine.

public static class Function1
{
    private static readonly ConcurrentQueue<IEnumerable<Guid>> updateQueue = new ConcurrentQueue<IEnumerable<Guid>>();

    public static async Task<IActionResult> UpdateRun(
        [HttpTrigger(AuthorizationLevel.Admin, "post", Route = null)] HttpRequest req, ExecutionContext execContext)
    {
        logger.LogInformation($"FunctionUpdate Start");
        using var reader = new StreamReader(req.Body);
        var request = JsonConvert.DeserializeObject<IEnumerable<Request>>(reader.ReadToEnd());
        var correlationIds = request?.Select(s => s.CorrelationId);

        updateQueue.Enqueue(correlationIds);
        return new OkObjectResult(new Response { HttpStatusCode = HttpStatusCode.Accepted });
    }

    [FunctionName("FunctionHandleQueue"), Timeout("00:05:00")]
    public static async Task HandleQueue([TimerTrigger("0 */1 * * * *")] TimerInfo myTimer, ExecutionContext execContext) // once every 1 minutes
    {
        logger.LogInformation($"before updateQueue condition : {updateQueue.Count}"); 
        if (updateQueue.Count > 0)
        {
            logger.LogInformation($"after updateQueue condition {updateQueue.Count}");

            var guids = new List<Guid>();
            var count = 0;
            while (count <= 1000 && updateQueue.Count > 0)
            {
                updateQueue.TryDequeue(out var updateRequest);
                var enumerable = updateRequest.ToList();
                count += enumerable.Count;
                guids.AddRange(enumerable);
            }

            await new ProcessUpdateSales(CreateMapper(), execContext)
                .Orchestrate(guids)
        }
    }
}

Logs are created when TimerTrigger executes every 1 minute:

before updateQueue condition : 0

Why is updateQueue.Count always 0? What am I doing wrong?

While Azure Functions may typically share a single backplane, it is not guaranteed. Resources can be spun down or up at any time and new copies of functions may not have access to the original state. If you need to preserve state, use a durable function or an actual queue, like an Azure Storage Queue. — David L
– David L, Commented Jan 15, 2021 at 20:55
To add to David's comment: it works locally on your machine because there's just a single local application domain spun up by dev runtime of Functions, so you add and remove from the same queue object. — Roman Polunin
– Roman Polunin, Commented Jan 15, 2021 at 21:01
Thank you David & Roman. So you means updateQueue will not guaranteed available while TimerTrigger function tries to access even though updateQueue is just specific to single function ? — tt0206
– tt0206, Commented Jan 15, 2021 at 21:50
Correct. A single function in code does not a single runtime function make :). — David L
– David L, Commented Jan 15, 2021 at 21:55

David L · Accepted Answer · 2021-01-19 18:48:31Z

While Azure Functions may typically share a single backplane, it is not guaranteed. Resources can be spun down or up at any time and new copies of functions may not have access to the original state. As a result, if you use static fields to share data across function executions, it should be able to reload the data from an external source.

That said, this is also not necessarily preferable due to how Azure Functions are designed to be used. Azure Functions enable high-throughput via dynamic scalability. As more resources are needed to process the current workload, they can be provisioned automatically to keep throughput high.

As a result, doing too much work in a single function execution can actually interfere with overall system throughput, since there is no way for the function backplane to provision additional workers to handle the load.

If you need to preserve state, use a form of permanent storage. This could take the form of a Azure Durable Function, an Azure Storage Queue, an Azure Service Bus queue, or even a database. In addition, in order to best take advantage of your function's scalability, try to reduce the workload to manageable batches that allow for large amounts of parallel processing. While you may need to frontload your work in a single operation, you want the subsequent processing to be more granular where possible.

Collectives™ on Stack Overflow

ConcurrentQueue dequeue problem in Azure function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related