How to execute Spark code locally with databricks-connect?

Question

Is there a way to execute Spark code locally with databricks-connect?

The reason is that I would like to execute some tests as part of my CI/CD pipeline without the need to have a cluster up and running.

I'm having a similar issue with this for unit-testing mocking spark_session. How you solved it? — Carleto
– Carleto, Commented Feb 24, 2021 at 17:58

Mike Wise · Accepted Answer · 2021-02-06 19:56:25Z

5

No, databricks-connect requires a running cluster. If you do not use any databricks specific code (like dbutils) you can run spark locally and execute against that - assuming you can still access the data sources you need.

edited Feb 6, 2021 at 19:56

Mike Wise

22.9k10 gold badges85 silver badges107 bronze badges

answered Jul 24, 2019 at 16:46

simon_dmorias

2,4933 gold badges23 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Flavi Over a year ago

Sad news as I will need to wait for DDL/DML available in Databricks to be supported in open source Delta Lake. Thanks.

Collectives™ on Stack Overflow

How to execute Spark code locally with databricks-connect?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related