Notebook path can't be in DBFS?

Question

Some of us are working with IDEs and trying to deploy notebooks (.py) files to dbfs. the problem I have noticed is when configuring jobs, those paths are not recognized.

notebook_path: If I use this :

  dbfs:/artifacts/client-state-vector/0.0.0/bootstrap.py. 


Only absolute paths are currently supported. Paths must begin with '/'."

If I use this;

/dbfs/artifacts/client-state-vector/0.0.0/bootstrap.py

or 
/artifacts/client-state-vector/0.0.0/bootstrap.py

I get Notebook not found.

what could be the issue here?

I see from the Databricks's architecture that Notebooks are in Microsoft managed subscription whereas DBFS is in Customer's subscription. Could that be the reason (that Notebook task is only able to pick from the microsoft managed subscription)? e.g. The folders I created on the workspace level where I have some notebooks, do not show up in the DBFS browser, so I am beginning to think that could be the reason.

Alex Ott · Accepted Answer · 2021-06-21 13:33:59Z

1

Notebooks aren't the files on file system - they are stored inside the control plane, not in the data plane where DBFS is located. If you want to execute notebook - you need to upload it via Workspace API - import, or via databricks workspace import ... command of databricks-cli or via databricks_notebook resource of Databricks Terraform provider. Only after that you will be able to refer it in the notebook_path parameter.

answered Jun 21, 2021 at 13:33

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Saugat Mukherjee Over a year ago

Yeah, that is what I suspected and wrote above. We uploaded it via the "Notebooks upload" task in Azure DevOps, earlier today and that made it work. Thanks for confirming.

Alex Ott Over a year ago

You may consider using Databricks Repos to sync code: docs.databricks.com/repos.html. I have a demo that showing how to use it as part of DevOps CI/CD: github.com/alexott/databricks-nutter-projects-demo. Also you may look onto github.com/databrickslabs/cicd-templates if you have more IDE-oriented workflows

Saugat Mukherjee Over a year ago

Thanks, will check them out. right now , we use Azure Devops to roll out to diff workspaces in diff environments. That includes standard devops tasks along with some powershell scripts with rest apis

Alex Ott Over a year ago

yeah, but REST API is cumbersome to use. You can replace it with databricks-cli, or use terraform provider for deployments

Collectives™ on Stack Overflow

Notebook path can't be in DBFS?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related