Scala error: value $ is not a member of object org.apache.spark.api.java.JavaSparkContext

Question

I'm new to Spark. I'm trying to run the following code in Spark shell:

import org.apache.spark.api.java.JavaSparkContext
import org.apache.hadoop.conf
JavaSparkContext context = new JavaSparkContext(conf);

But I'm getting the following error:

<console>:32: error: value context is not a member of object 
org.apache.spark.api.java.JavaSparkContext
val $ires10 = JavaSparkContext.context
                           ^
<console>:29: error: value context is not a member of object 
org.apache.spark.api.java.JavaSparkContext
     JavaSparkContext context = new JavaSparkContext(conf);

Is there any import statement that I'm missing? I also added

import org.apache.spark.api.java.JavaSparkContext._

but it still didn't work. Please help.

Mike Allen · Accepted Answer · 2017-10-19 18:37:28Z

2

UPDATE: Whether the configuration is valid or not is something you'll have to work on, but this addresses the error you asked about in your original question.

Your code is (almost) valid Java, but not valid Scala. Did you mean something like this:

import org.apache.spark.api.java.JavaSparkContext

val context = new JavaSparkContext()

Alternatively, since you're using Scala, you might want to try this instead.

import org.apache.spark.SparkContext

val context = new SparkContext()

As for the error you report, Scala will treat the statement JavaSparkContext context as a reference to a member named context of an object named JavaSparkContext - and not as a variable declaration as it is in Java.

edited Oct 19, 2017 at 18:37

answered Oct 19, 2017 at 16:18

Mike Allen

8,3773 gold badges26 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Non Volatile Engineer Over a year ago

I did try that earlier, but I got an error saying <console>:38: error: package org.apache.hadoop.conf is not a value

Mike Allen Over a year ago

That's probably because org.apache.hadoop.conf is a package, not a configuration instance. You need to determine how to create the object that you pass to JavaSparkContext's constructor.

Collectives™ on Stack Overflow

Scala error: value $ is not a member of object org.apache.spark.api.java.JavaSparkContext

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related