error spark-submit scala, ClassNotFoundException

Question

I'm trying to run a scala application and in spark-shell this works well. But when I use spark-submit, using my class, it fails.

spark-submit --deploy-mode cluster --master yarn --class org.apache.spark.examples.SparkPi s3n://bucket/test.scala

Applicacion:

package org.apache.spark.examples

import org.apache.spark.sql.types._
import org.apache.spark.sql.SQLContext

object SparkPi {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application")
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)

    print("test")

  }
}

Error:

Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi)

Zyoma · Accepted Answer · 2016-09-15 11:48:25Z

1

Try to build jar using your test.scala source and provide it as an argument for spark-submit. In spark-submit you should specify a jar with your compiled code, not the source code itself.

answered Sep 15, 2016 at 11:48

Zyoma

1,5781 gold badge10 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

David Over a year ago

I compiled with SBT, no?

Zyoma Over a year ago

Yep, and got some .class files. Now pack them into a jar and use it to launch your job. According to a command that you provided you're specifying a source code .scala instead of compiled code.

Collectives™ on Stack Overflow

error spark-submit scala, ClassNotFoundException

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related