Azure Function Trigger for List<POCO> Storage Queue

Question

We have a subscription to Event Grid with an endpoint type as Storage Queue (say xyz-queue). We also have an Azure Function which is a Queue (xyz-queue) trigger. Every message in xyz-queue storage Queue is EventGridEvent object. I have Azure Function trigger something like below:

public static void Run([QueueTrigger(xyz-queue, Connection = "connnection")] EventGridEvent ege)

Since, there will be 1000's of messages with same POCO object (in this case EventGridEvent), is there a way where we can read multiple messages, process them together and also ensure we are not reading more than 5 messages so that function do not take long time to process.

Joey Cai · Accepted Answer · 2020-07-15 07:35:45Z

1

The number of queue messages that the Functions runtime retrieves simultaneously and processes in parallel. When the number being processed gets down to the newBatchThreshold, the runtime gets another batch and starts processing those messages. So the maximum number of concurrent messages being processed per function is batchSize plus newBatchThreshold. This limit applies separately to each queue-triggered function.

{
    "version": "2.0",
    "extensions": {
        "queues": {
            "maxPollingInterval": "00:00:02",
            "visibilityTimeout" : "00:00:30",
            "batchSize": 5,
            "maxDequeueCount": 5,
            "newBatchThreshold": 2
        }
    }
}

For more details, you could refer to this article.

Also, if you want to output queue message multiple with poco, you could use ICollector and IAsyncCollector.

answered Jul 15, 2020 at 7:35

Joey Cai

20.2k1 gold badge25 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Vicky Over a year ago

So by default the batch size is 16. is it like each function instance can read max 1 message from the queue at a time? So there will be 16 function instances processing 16 messages from queue concurrently in a VM?

Joey Cai Over a year ago

No. You can process 16 message in parallel in one instance.

Ross Sep 12 at 22:06

Note that the minimum allowed "Maximum instance count" is 40, meaning you can't prevent Azure from running less than 40 concurrent instances if it wants to. That can cause problems if they all have a shared dependency (e.g. if all 40 slam the same database); design your solution accordingly.

Collectives™ on Stack Overflow

Azure Function Trigger for List<POCO> Storage Queue

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related