You cannot run a notebook job on a cluster running Databricks Basic

Question

Error code
3201
Troubleshooting guide
Failure type
User configuration issue
Details
You cannot run a notebook job on a cluster running Databricks Basic.
Source

I am gettig this error when i try to run my Databrick notebook in ADF. I am not understanding what this means. I tried to create a different type of cluster, but still failing due to this error.

Can Azure guru please help me with this.

@PratikSomaiya yes, I am able to run the script from azure Databrick workspace. No issue at all. Now I am getting a different error this time. It is saying I am exceeding the core limit for my region. I am living in New Zealand and my closest data center is Australia East. But it is still running well from Azure Databrick workstation, but from Azure Data Factory. The error messages are confusing me as I am not even running a big chunk of script. — user86907
– user86907, Commented Mar 19, 2022 at 7:46
ok got it, I haven't seen such an error as all Databricks notebook activity that I used were successful, can you try upgrading to a Standard tier once? — Pratik Somaiya
– Pratik Somaiya, Commented Mar 19, 2022 at 10:01
Azure subscriptions have a CPU core quota limit which restricts the number of CPU cores you can use. This is a hard limit. See these articles from Microsoft for details - learn.microsoft.com/en-us/azure/databricks/kb/clusters/…, learn.microsoft.com/en-us/azure/azure-resource-manager/…, learn.microsoft.com/en-us/azure/databricks/kb/clusters/… — Dipanjan Mallick
– Dipanjan Mallick, Commented Mar 20, 2022 at 15:06
@DKNY aha, that is probably why then. I am on a free trial subscription, and Microsoft is probably limiting the number of CPU cores I can use with my subscription. I probably need to upgrade my subscription. Does that mean I have to be on a pay as you go subscription though? — user86907
– user86907, Commented Mar 21, 2022 at 3:41

Abhishek Khandave · Accepted Answer · 2022-03-21 12:32:19Z

1

You can post your comment as answer and accept it. I have mentioned below how you can upgrade your subscription and resolve this issue.

To upgrade you need to select Standard or Premium Pricing Tier while creating resource.

answered Mar 21, 2022 at 12:32

Abhishek Khandave

3,2621 gold badge9 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user86907 · Accepted Answer · 2022-03-21 21:57:01Z

1

I am on a free trial subscription, and Microsoft is probably limiting the number of CPU cores I can use with my current subscription. I need to upgrade my subscription.

answered Mar 21, 2022 at 21:57

user86907

85711 silver badges26 bronze badges

Collectives™ on Stack Overflow

You cannot run a notebook job on a cluster running Databricks Basic

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related