Azure Durable Functions' CallActivityWithRetryAsync does not retry on failure

Question

In my orchestrator function, I upload a request file to an external server. After some non-deterministic amount of time, a response file should have been generated. I need to poll for this file and download it.

My current approach is to wait 10mins after upload. And then use the built-in CallActivityWithRetryAsync with RetryOptions. After the first poll/download failure, wait 5mins before starting a total of 10 retry attempts. A retry should only be attempted if an exception with the message RESPONSE_FILE_NOT_YET_AVAILABLE is thrown within the activity function.

        var nowIn10Minutes = ctx.CurrentUtcDateTime.AddMinutes(10);
        await ctx.CreateTimer(nowIn10Minutes, CancellationToken.None);

        const string RETRY_ERROR_MESSAGE = "RESPONSE_FILE_NOT_YET_AVAILABLE";
        var retryOptions = new RetryOptions(TimeSpan.FromMinutes(5), 10)
        {
            Handle = ex => ex.Message == RETRY_ERROR_MESSAGE
        };
        await ctx.CallActivityWithRetryAsync(nameof(PollForResponseAndDownload), retryOptions, input);

However, according to the logs, this retry logic is not honoured. See below.

After waiting 10mins as set in the timer, the orchestration immediately fails with a FunctionFailedException. No retry is executed, though the correct exception message is shown in the logs.

Am I fundamentally misunderstanding the process? Here are the relevant logs:

-> After uploading the request, wait 10mins

2022-01-31 00:00:06.740 <GUID>: Function 'MyOrchestrator (Orchestrator)' is waiting for input. Reason: CreateTimer:2022-01-31T00:10:06.5093237Z. IsReplay: False. State: Listening. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 112.
2022-01-31 00:00:06.741 <GUID>: Function 'MyOrchestrator (Orchestrator)' awaited. IsReplay: False. State: Awaited. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 113.

-> Resume after 10mins, schedule activity function for execution

2022-01-31 00:10:32.700 <GUID>: Function 'MyOrchestrator (Orchestrator)' was resumed by a timer scheduled for '2022-01-31T00:10:06.5093237Z'. IsReplay: False. State: TimerExpired. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 114.
2022-01-31 00:10:32.701 <GUID>: Function 'PollForResponseAndDownload (Activity)' scheduled. Reason: MyOrchestrator. IsReplay: False. State: Scheduled. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 115.
2022-01-31 00:10:32.701 <GUID>: Function 'MyOrchestrator (Orchestrator)' awaited. IsReplay: False. State: Awaited. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 116.

-> Start activity function. It fails immediately with expected ex.Message, but still fails to run retry logic.

2022-01-31 00:10:32.715 <GUID>: Function 'PollForResponseAndDownload (Activity)' started. IsReplay: False. Input: (368 bytes). State: Started. HubName: <HUB-NAME>. AppName: <APP-NAME>. SlotName: Production. ExtensionVersion: 2.6.0. SequenceNumber: 117. TaskEventId: 5
2022-01-31 00:10:37.078 <GUID>: Function 'PollForResponseAndDownload (Activity)' failed with an error. Reason: System.Exception: RESPONSE_FILE_NOT_YET_AVAILABLE at MyNamespace.func._getResponseFileContents(String fileHeader) in C:\Users\me\source\AppName\func.cs:line ...
2022-01-31 00:10:37.364 <GUID>: Function 'MyOrchestrator (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'PollForResponseAndDownload' failed: "RESPONSE_FILE_NOT_YET_AVAILABLE". See the function execution logs for additional details. ---> System.Exception: RESPONSE_FILE_NOT_YET_AVAILABLE at ...

Bjorne · Accepted Answer · 2022-01-31 11:34:26Z

2

The activity function adds "Activity function 'SomeActivityFunc' failed: " to the message. So either create a custom exception type to be thrown and check type, use .Contains or check for "Activity function 'SomeActivityFunc' failed: RESPONSE_FILE_NOT_YET_AVAILABLE" instead.

Handle = ex => ex.Message == "Activity function 'SomeActivityFunc' failed: " + RETRY_ERROR_MESSAGE
Handle = ex => ex.Message.Contains(RETRY_ERROR_MESSAGE)
Handle = ex => ex is SomeCustomExceptionType

answered Jan 31, 2022 at 11:34

Bjorne

1,4999 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

SaiSakethGuduru · Accepted Answer · 2022-01-31 11:19:58Z

0

Here are the few links with related discussions. Can you try rechecking your function app according to this link for resolving your issue.

When the CallActivityWithRetryAsync call is made the DurableOrchestrationContext calls the ScheduleWithRetry method of the OrchestrationContext class inside the DurableTask framework.
There the Invoke method on the RetryInterceptor class is called and that does a foreach loop over the maximum number of retries. This class does not expose properties or methods to obtain the number of retries.

answered Jan 31, 2022 at 11:19

SaiSakethGuduru

2,4751 gold badge8 silver badges17 bronze badges

Collectives™ on Stack Overflow

Azure Durable Functions' CallActivityWithRetryAsync does not retry on failure

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related