Scala: Spark: java.lang.ClassNotFoundException:

Question

I try make Apache Spark job on scala. I'm novice in Scala and and earlier use Pyspark. Have error when the job starts. Code:

object SparkRMSP_full {
  import org.apache.spark.sql.SparkSession

  def main(args: Array[String]): Unit = {
    val spark = SparkSession
      .builder
      .appName("parse_full_rmsp_job")
      .getOrCreate()

    val raw_data_df = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "10.1.24.111:9092")
      .option("subscribe", "dev.etl.fns.rmsp.raw-data")
      .load()

    println(raw_data_df.isStreaming)
    raw_data_df.printSchema
  }
}

spark-submit command:

spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.1.0 --master local --num-executors 2 --executor-memory 2g --driver-memory 1g --executor-cores 2 "C:\tools\jar\streaming_spark.jar"

And I have error:

20/07/15 15:05:32 WARN SparkSubmit$$anon$2: Failed to load SparkRMSP_full.
java.lang.ClassNotFoundException: SparkRMSP_full

How I must declare the class correctly?

UPD:

build.sbt:

name := "streaming_spark"

version := "0.1"

scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.1"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10-assembly" % "2.3.1"

project sctructure on pastebin

how did you create a jar file, maven or sbt can you share it? Also add --class SparkRMSP_full — koiralo
– koiralo, Commented Jul 16, 2020 at 8:52
I'm build jar file with sbt in IDEA. And I add --class into spark-submit, but same error — Антон Букреев
– Антон Букреев, Commented Jul 16, 2020 at 10:18

s.polam · Accepted Answer · 2020-07-16 08:52:13Z

1

Change your spark-submit command like below & try again.

spark-submit \
  --packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.1.0 \
  --master local \
  --num-executors 2 \
  --executor-memory 2g \
  --driver-memory 1g \
  --executor-cores 2 \
  --class SparkRMSP_full \  # you might need add your fully qualified package name with class name
  "C:\tools\jar\streaming_spark.jar"

answered Jul 16, 2020 at 8:52

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

s.polam Over a year ago

which build tool are you using maven or sbt .. can you post that code in question ??

Антон Букреев Over a year ago

what I must post? I novice in Java-like codding. I run "Build" in IDEA and make jar.

s.polam Over a year ago

pom.xml or build.sbt file ? and how are you builiding jar file ?

Антон Букреев Over a year ago

I'm build jar file with sbt in IDEA. build.sbt in question

s.polam Over a year ago

try building fat jar file using sbt also include main class.

Collectives™ on Stack Overflow

Scala: Spark: java.lang.ClassNotFoundException:

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related