How to run .Net spark jobs on Databricks from Azure Data Factory?

Question

In Azure data factory, you have a Databricks Acvitiy. This activity supports running python, jar and notebooks. And These notebooks may be written in scala, python, java, and R but not c#/.net.

Is there inherent or direct support where I can write my .NET spark code and run it on Databricks from Data Factory?

Can I use .NET spark in Azure Databricks to its full extent?

Have you referred to learn.microsoft.com/en-us/dotnet/spark/tutorials/…? — Jim Xu
– Jim Xu, Commented Aug 5, 2020 at 9:10
Yes I have. But as you can see this is customised and uses CLI. As I mentioned I have also done the same. What I want to know is that in any way can I invoke .NET spark on bricks using Azure data factory — Nikhil Sharma
– Nikhil Sharma, Commented Aug 5, 2020 at 10:25

Ed Elliott · Accepted Answer · 2020-09-09 12:07:48Z

2

You specify that you want to launch a JAR file. The .NET for Apache Spark project uses a JAR file to start a listener which the .NET code then connects to.

The JAR is microsoft-spark-2.4.x-0.12.1.jar (depends on the version of spark and the .NET NuGet version). The Class to run is org.apache.spark.deploy.dotnet.DotnetRunner which you need to pass the correct parameters to, so that it starts your .NET application.

Ed

answered Sep 9, 2020 at 12:07

Ed Elliott

6,88420 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to run .Net spark jobs on Databricks from Azure Data Factory?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related