java.lang.NoClassDefFoundError: better/files/File in spark-sbumit for scala code

Question

When i export the program into a jar file and execute it i get an java.lang.NoClassDefFoundError: better/files/File error.

the code i'm using is below.

Thanks in advance for any assistance

SBT

name := "testFunctions"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "com.github.pathikrit" %% "better-files" % "2.17.1"
libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.1.0",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.0",
  "com.github.pathikrit" %% "better-files" % "2.17.1"
)

initialize := {
  val _ = initialize.value
  if (sys.props("java.specification.version") != "1.8")
    sys.error("Java 8 is required for this project.")
}

Scala Code

/**
  * Created by cloudera on 7/23/17.
  */

import better.files.File._
import org.apache.spark.sql.SparkSession


object funcJM {


  val forDelete = (root/"/home/cloudera/Documents/fabo")
    .createIfNotExists()

  if (forDelete.exists)
    forDelete.delete()


  def main(args:Array[String]) : Unit = {

    val spark = SparkSession.builder
      .master("local")
      .appName("Get ForEx Data")
      .getOrCreate()



  }

}

Command line executing jar file

spark-submit --class funcJM --master local[*] /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar --driver-class-path /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar

Error

Exception in thread "main" java.lang.NoClassDefFoundError: better/files/File at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:722) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: better.files.File at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 10 more

The Tree structure where the code live looks like below - so the class called func lives in the scala directory

\main
    \java
    \resources
    \scala
        -funcJM(class)
    \scala-2.11

an simple alternative to the fat jar is described here: stackoverflow.com/q/51387919/666886 — philwalk
– philwalk, Commented Jul 17, 2018 at 18:26

wllmtrng · Accepted Answer · 2017-07-27 05:34:38Z

2

There are three things you can do:

Use something like sbt assembly to compile a fat jar. The jar that includes better.files.File will be packaged in testfunctions_2.11-1.0.jar.
Copy the jar file that includes the better package into the spark jars directory: ../spark-2.1.0-bin-hadoop2.7/jars
Specify the jar file that includes the better package as an argument in your spark-submit call using --driver-class-path better-files-akka_2.10.jar

answered Jul 27, 2017 at 5:34

wllmtrng

1823 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

java.lang.NoClassDefFoundError: better/files/File in spark-sbumit for scala code

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related