ClassNotFoundException when running Spark application with spark-submit

Question

I am not sure how to launch a Java app with spark-submit.

When I run the following command:

spark@mcava-master:/home/miren/NetBeansProjects$ /opt/spark/bin/spark-submit --class /home/miren/NetBeansProjects/SparkExample/src/main/java/com/mycompany/sparkexample/CountWords.java --master spark://192.168.1.105:7077 /home/miren/NetBeansProjects/SparkExample/target/SparkExample-1.0-SNAPSHOT.jar spark://192.168.1.105:7077

I get an exception:

java.lang.ClassNotFoundException: /home/miren/NetBeansProjects/SparkExample/src/main/java/com/mycompany/sparkexample/CountWords.java
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

How do I indicate the classpath?

@Miren: FYI, this question is being discussed on Meta Stack Overflow. — S.L. Barth is on codidact.com
– S.L. Barth is on codidact.com, Commented Feb 23, 2016 at 9:37

Markon · Accepted Answer · 2017-06-02 09:27:02Z

11

You need to call spark-submit in the following way:

spark-submit --class com.mycompany.sparkexample.CountWords 
             --master spark://192.168.1.105:7077 
             /home/miren/NetBeansProjects/SparkExample/target/SparkExample-1.0-SNAPSHOT.jar 
             {your-app-arguments-here}

The --class parameter takes the fully qualified name of the object containing your main method.

The documentation contains more information about using spark-submit properly.

edited Jun 2, 2017 at 9:27

answered Feb 22, 2016 at 10:13

Markon

4,6101 gold badge29 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Markon Over a year ago

You should definitely have a look at spark.apache.org/docs/latest/submitting-applications.html. This explains how to use spark-submit properly :)

Boris the Spider Over a year ago

@Miren if this answered your question consider accepting the answer.

Collectives™ on Stack Overflow

ClassNotFoundException when running Spark application with spark-submit

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related