sbt error: object spark is not a member of package org.apache

Question

I installed sbt-1.3.4.msi and when trying to build a sample SparkPi.scala app, I'm getting the following error:

C:\myapps\sbt\sparksample>sbt
[info] Loading project definition from C:\myapps\sbt\sparksample\project
[info] Compiling 1 Scala source to C:\myapps\sbt\sparksample\project\target\scala-2.12\sbt-1.0\classes ...
[error] C:\myapps\sbt\sparksample\project\src\main\scala\SparkPi.scala:3:19: object spark is not a member of package org.apache
[error] import org.apache.spark._
[error]                   ^
[error] C:\myapps\sbt\sparksample\project\src\main\scala\SparkPi.scala:8:20: not found: type SparkConf
[error]     val conf = new SparkConf().setAppName("Spark Pi")
[error]                    ^
[error] C:\myapps\sbt\sparksample\project\src\main\scala\SparkPi.scala:9:21: not found: type SparkContext
[error]     val spark = new SparkContext(conf)
[error]                     ^
[error] three errors found
[error] (Compile / compileIncremental) Compilation failed

The SparkPi.scala file is in C:\myapps\sbt\sparksample\project\src\main\scala (as shown in the error messages above).

What am I missing here?

The C:\myapps\sbt\sparksample\sparksample.sbt file is as follows:

name := "Spark Sample"

version := "1.0"

scalaVersion := "2.12.10"

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.0"

Jacek Laskowski · Accepted Answer · 2020-07-07 18:49:24Z

1

C:\myapps\sbt\sparksample\project\src\main\scala directory has SparkPi.scala file

That's the problem. You've got the Scala file(s) under project directory that's owned by sbt itself (not your sbt-managed Scala project).

Move the SparkPi.scala and other Scala files to C:\myapps\sbt\sparksample\src\main\scala.

answered Jul 7, 2020 at 18:49

Jacek Laskowski

75k28 gold badges253 silver badges440 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

frosty Over a year ago

After above fix, it ran without error, but there is no .jar file in C:\myapps\sbt\sparksample\target. Instead, I see a file called "out" in C:\myapps\sbt\sparksample\target\streams_global\ivyConfiguration_global\streams.

frosty Over a year ago

The issue is fixed by changing the sbt file's line to libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.0"

Jacek Laskowski Over a year ago

@frosty Thanks for accepting my answer! Much appreciated. As for your "fix", I'd disagree, i.e. += operator is for a single dependency while ++= is for zero, one or more dependencies. They're equivalent in your case. BTW, You don't need to look at C:\myapps\sbt\sparksample\target\streams_global directory since it's internal to sbt (not for users).

Collectives™ on Stack Overflow

sbt error: object spark is not a member of package org.apache

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related