Spark program cannot open SQL Server context?

Question

In the file main.scala,

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import java.sql._

object ConnTest extends App {
  val conf = new SparkConf()
  val sc = new SparkContext(conf.setAppName("Test").setMaster("local[*]"))
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)

  sc.stop()
}

However, the sbt run got the following error.

[info] Compiling 1 Scala source to C:\Users\user1\IdeaProjects\sqlServer\target\scala-2.11\classes...
[error] C:\Users\user1\IdeaProjects\sqlServer\src\main\scala\main.scala:9: type SQLContext is not a member of package org.apache.spark.sql
[error]   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
[error]                                             ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed May 11, 2017 6:11:47 PM

First, what are your dependencies - sql must be explicitly set? Also you shouldn't use sbt run for spark, but spark-submit instead. — Justin Pihony
– Justin Pihony, Commented May 11, 2017 at 23:38
I should use SBT compile the code first. Maybe I shouldn't use "run"? — ca9163d9
– ca9163d9, Commented May 12, 2017 at 1:11

koiralo · Accepted Answer · 2017-05-12 02:23:49Z

1

You should use Spark 2 which has single entry point SparkSession. you can create SQLContext and SparkContext as

val sparkSession = SparkSession.builder().master("local[*]").getOrCreate()

val sc = sparkSession.sparkContext
val sqlC = sparkSession.sqlContext

Include the dependency for spark core and spark sql

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.1"

answered May 12, 2017 at 2:23

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ca9163d9 Over a year ago

Do I need to download and jdbc driver for SQL server?

Peter Halverson Over a year ago

Spark does not include any JDBC drivers, you are responsible for providing them in the application bundle or other classpath item.

ca9163d9 Over a year ago

@ShankarKoirala No I'm reading data from Microsoft Sql Server.

koiralo Over a year ago

You need to add and bundle the jar yourself. you can also provide the jar during spark-submit

ashburshui · Accepted Answer · 2017-05-12 00:19:08Z

Your class path is correct, please carefully check you maven or sbt configure

In maven, you should add this specific dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.1.1</version>
    <scope>compile</scope>
</dependency>

sbt is similar too, if you want to add this dependency at compile time and package an uber jar with it.

But sometime the spark-core jar is provided by the cluster at runtime, if it is in this case, you can adjust the scope value at you convenient, maybe provided.

What must to be said is that sbt or maven is just dependency control tool, not relevant to spark's execution, you should get your jar package upload on the cluster, and use the spark-submit program to run it!

Please see the community document of the spark example, and have a try:

http://spark.apache.org/examples.html

Good Luck!

Collectives™ on Stack Overflow

Spark program cannot open SQL Server context?

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related