Currently I am learning how to use databricks-connect to develop Scala code using IDE locally. The set-up of the databricks-connect as described here was succuesful. This is an assumption as I can run PySpark Code against the cluster with no issues.
However, I struggle to run the scala example as provided in the Scala demo object. I receive the java.lang.ClassNotFoundException: Here the full trace of the Exception:
My build.sbt follows closely what was provided with the Databricks-connect demo:
scalaVersion := "2.12.15"
name := "hello-world"
organization := "ch.epfl.scala"
version := "1.0"
//autoScalaLibrary := true
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "2.1.1"
unmanagedBase := new java.io.File("/home/clh78p/.local/lib/python3.8/site-packages/pyspark/jars")
mainClass := Some("Main")
My clusters configuration is up-date and I payed close attention to following the databricks-connect client installation manual precisely.
I read amongst others the post here and this post. They make me think I am missing some jar or a library dependency which may not be covered in the demo? I made sure the scala versions match, so this should hopefully not be the problem. Any advice is welcome.
Note that I use VS Code with WSL2 and the Databricks Azure option. I am learning Scala at the moment to replace PySpark Notebooks.
