How to setup and run scala-spark in intellij?

Question

I am trying to run use Intellij to build spark applications written in scala. I get the following error when I execute the scala program:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
    at Main$.main(Main.scala:10)
    at Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 2 more

the line throwing the error is the following one:

val conf = new SparkConf().setAppName("Sample Spark Scala Application")

I am not getting any error if I just try to import spark SparkConf() without trying to execute the above line.

The following are the contents of my sbt file:

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.12.15"

val sparkVersion = "3.2.4"

// Note the dependencies are provided
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided"

lazy val root = (project in file("."))
  .settings(
    name := "untitled4"
  )

The versions mentioned in the sbt match with what I can see when I open the spark shell.

the following are the contents of my PATH, JAVA_HOME and SPARK_HOME variables

ray@Rayquaza-ASUS-TUF-Gaming-F15-FX506HEB-FX566HEB:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/spark/spark-3.2.4-bin-hadoop2.7/bin
ray@Rayquaza-ASUS-TUF-Gaming-F15-FX506HEB-FX566HEB:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64
ray@Rayquaza-ASUS-TUF-Gaming-F15-FX506HEB-FX566HEB:~$ echo $SPARK_HOME
/usr/spark/spark-3.2.4-bin-hadoop2.7

I am able to run code properly in the spark shell. So spark and scala seem to be properly setup. But Intellij isn't able to use those.

I tried following a few online guides but they weren't helpful. Would appreciate help solving the problem. Also let me know if any more information or details are required.

ripone07 · Accepted Answer · 2024-01-10 16:11:19Z

1

If you want to run the code from inside Intellij, the spark dependencies are not "provided". Running through Intellij with default configuration, your application is going to run as a simple JVM application and should be provided all its dependencies in the classpath. Something like the below is happening by default when you press the Run button in a JVM project:

java -classpath /path/to/dependency:/path/to/dependency_2 YourMain

So, my guess is that when your run the code from Intellij because your spark dependencies are marked as provided are nowhere to be found by java in the provided classpath. You should change the scope of the spark packages to compile in your sbt file, build the project and try again.

answered Jan 10, 2024 at 16:11

ripone07

416 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

PRATIK CHAPADGAONKAR Over a year ago

Tried out your solution. Sadly it didn't work

ripone07 Over a year ago

Do you get the same error?

PRATIK CHAPADGAONKAR Over a year ago

I invalidated the cache and restarted the application. The error was fixed after that

Anand Satheesh · Accepted Answer · 2024-01-11 03:35:17Z

0

Here is the sbt and folder structure of a sample spark scala code I ran in IntelliJ with the build.sbt contents and logs of execution

answered Jan 11, 2024 at 3:35

Anand Satheesh

1241 silver badge6 bronze badges

Comments

Mayur Surani · Accepted Answer · 2024-08-15 12:56:30Z

0

If you follow spark dependencies as per attached image, should work out.

answered Aug 15, 2024 at 12:56

Mayur Surani

132 bronze badges

1 Comment

Algamest Over a year ago

Please share any relevant code instead of a screenshot by editing your Answer.

Collectives™ on Stack Overflow

How to setup and run scala-spark in intellij?

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related