spark-cassandra (java.lang.NoClassDefFoundError: org/apache/spark/sql/cassandra/package)

Question

i am trying to read DataFrame from cassandra 4.0.3 with spark 3.2.1 using scala 2.12.5 and sbt 1.6.2 but i have a problem.

this is my sbt file:

name := "StreamHandler"

version := "1.6.2"

scalaVersion := "2.12.15"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "3.2.1" % "provided",
    "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
    "org.apache.cassandra" % "cassandra-all" % "4.0.3" % "test",
    "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided",
    "com.datastax.spark" %% "spark-cassandra-connector" % "3.2.0",
    "com.datastax.cassandra" % "cassandra-driver-core" % "4.0.0" 
)
libraryDependencies += "com.datastax.dse" % "dse-java-driver-core" % "2.1.1" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided"
libraryDependencies += "org.apache.commons" % "commons-math3" % "3.6.1" % "provided"

and this is my scala file:

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming._
import org.apache.spark.sql.types._
import org.apache.spark.sql.cassandra._

import com.datastax.oss.driver.api.core.uuid.Uuids
import com.datastax.spark.connector._


object StreamHandler {
    def main(args: Array[String]){

        val spark = SparkSession
            .builder
            .appName("Stream Handler")
            .config("spark.cassandra.connection.host","localhost")
            .getOrCreate()

        import spark.implicits._


        val Temp_DF = spark
            .read
            .cassandraFormat("train_temperature", "project")
            .load()
        
        Temp_DF.show(10)
    }
}

and this is the result:

Alex Ott · Accepted Answer · 2022-05-10 18:34:01Z

1

Usually the problem is that when you do sbt package it builds a jar only with your code, and without dependencies. To mitigate this problem you have two approaches:

Specify Cassandra Connector when using spark-submit with --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 as described in documentation
Create a fat jar (with all necessary dependencies) using SBT assembly plugin, but you may need to take care of not including Spark classes into fat jar.

answered May 10, 2022 at 18:34

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

spark-cassandra (java.lang.NoClassDefFoundError: org/apache/spark/sql/cassandra/package)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related