How to get Spark in Java working - Could not initialize class org.apache.spark.util.Utils$

Question

I am attempting to connect to a standalone spark server from a java application using the following code

SparkConf sparkConf_new = new SparkConf()
    .setAppName("Example Spark App")
    .setMaster("spark://my.server.com:7077");
JavaSparkContext sparkContext = new JavaSparkContext(sparkConf_new);
JavaRDD<String> stringJavaRDD = sparkContext.textFile("hdfs://cluster/my/path/test.csv");
out.println("Number of lines in file = " + stringJavaRDD.count());

I am receiving the following error

An exception occurred at line 12

12: SparkConf sparkConf_new = new SparkConf()
13:     .setAppName("Example Spark App")
14:     .setMaster("spark://my.server.com:7077");
15: JavaSparkContext sparkContext = new JavaSparkContext(sparkConf_new);
16: JavaRDD<String> stringJavaRDD = sparkContext.textFile("hdfs://cluster/my/path/test.csv");
17: out.println("Number of lines in file = " + stringJavaRDD.count());

java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.util.Utils$
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:123)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:54)

Included are:

scala-library-2.10.5.jar
spark-core_2.10-1.6.0.jar
hadoop-core-1.2.1.jar

You're missing the original cause of your exception, which should be a previous ExceptionIninitializerError for the Util$ class. — Marko Topolnik
– Marko Topolnik, Commented Nov 23, 2016 at 14:02
that is the reasson I'm here; no where in my logs does that exist which is why I've asked, normally I would see this. @Nirmal Ram backend Spark 1.6.0 and Hadoop 2.7 — JRhino
– JRhino, Commented Nov 23, 2016 at 14:22
Just noticed the hadoop core version updated that to 1.2.1 same issue though — JRhino
– JRhino, Commented Nov 23, 2016 at 14:34
On the server you are connecting to can you open the spark-shell successfully? — Brad
– Brad, Commented Nov 23, 2016 at 19:44

Brad · Accepted Answer · 2016-11-23 19:50:05Z

1

You typically package your application into an Uber JAR file and use $SPARK_HOME/bin/spark-submit script to send it to the server for execution.

If you can try creating the most simple applicaiton to start with, using Maven all you should need in your project dependencies is

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
</dependency>

Doing it this way all of your environmental config (server url, etc) can be defined out side of your Java code in a script making it more portable.

edited Nov 23, 2016 at 19:50

answered Nov 23, 2016 at 19:42

Brad

16k11 gold badges62 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mariusz · Accepted Answer · 2016-11-23 16:49:57Z

0

If you write application in spark, even if you are sending it to remote cluster the three jars are not enough. You should add all spark dependencies to the classpath of the application.

The easiest way is to use maven or gradle (see http://spark.apache.org/docs/1.6.3/programming-guide.html#linking-with-spark) which will include spark and all its transitive dependencies. If you cannot use build system, download an official spark build and add all jars in jars/ directory to the classpath of your application.

answered Nov 23, 2016 at 16:49

Mariusz

14k3 gold badges66 silver badges66 bronze badges

1 Comment

Nirmal Ram Over a year ago

Plus hadoop core should be replaced with hadoop common jar

Collectives™ on Stack Overflow

How to get Spark in Java working - Could not initialize class org.apache.spark.util.Utils$

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related