Azure data factory data flow test connection Spark job failes

Question

Im using Linked service to connect the Azure SQL database to the Azure data factory data flow.

In the Azure data factory Linked service connection gets established successfully.

However, when I add my source dataset using a linked service to the data flow, there the test connection fails.

Below are the error messages:

Spark job failed: {
"text/plain": "{\"runId\":\"94405efb-4fd6-449e-aa7f-e4396f1db87f\",\"sessionId\":\"93dd501b-ac03-4c93-8773-36058e06b8a0\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"com.microsoft.dataflow.broker.MissingRequiredPropertyException: server is a required property for AzureSqlDatabase1.\\ncom.microsoft.dataflow.broker.PropertyNotFoundException: Could not extract value from AzureSqlDatabase1\",\"detailedMessage\":\"Failure 2024-05-17 07:10:36.828 failed DebugManager.processJob, run=94405efb-4fd6-449e-aa7f-e4396f1db87f, errorMessage=com.microsoft.dataflow.broker.MissingRequiredPropertyException: server is a required property for AzureSqlDatabase1.\\ncom.microsoft.dataflow.broker.PropertyNotFoundException: Could not extract value from AzureSqlDatabase1\"}}\n"
} - RunId: 94405efb-4fd6-449e-aa7f-e4396f1db87f

When I googled the above error most of the forums said to recreate the Linked service. In this StackOverflow thread it asks to increase the compute size of the Data Flow Integration which also does not work.

Is there any better solutions, or what exactly cause this issue?

Are you using azure sql database? or sql server in some azure vm? — Aswin
– Aswin, Commented May 20, 2024 at 7:03
I am not sure why this happens. I did multiple times. I am not able to reproduce the same issue and unable to find RC for this. — Aswin
– Aswin, Commented May 22, 2024 at 9:06
@Aswin whats your ADF region? . I tried deleting all my services and recreating them but still the same issue exists — Indrajith Ekanayake
– Indrajith Ekanayake, Commented May 22, 2024 at 9:54

NNPP · Accepted Answer · 2024-05-22 20:27:21Z

4

+50

Answer recommended by Microsoft Azure Collective

I had to create a new Linked Service, this time using the "legacy" version and not the "recommended". Add my server name and db name and test. Then publish the changes with that new LS. (I might have been able to use the existing, so long as I published after) Then change my sources to use that new Linked Service (I was getting the same error as you but in CDC using the Linked Service direct). I then no longer got that error. I think there might be a bug (seen similar posts on MS) and using Legacy in the Linked Service avoids it Here is my Linked Service config using Legacy

edited May 22, 2024 at 20:27

answered May 22, 2024 at 20:23

NNPP

1062 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Indrajith Ekanayake Over a year ago

Oh this works, thanks a lot. I was struggling with this for a while now.

Pallavi D Over a year ago

Same here, Just by changing the version from recommended to legacy in the linked service resolved this issue for me.

Collectives™ on Stack Overflow

Azure data factory data flow test connection Spark job failes

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related