Spark Scala : Unable to import sqlContext.implicits._

Question

I tried the below code and cannot import sqlContext.implicits._ - it throws an error (in the Scala IDE), unable to build the code:

value implicits is not a member of org.apache.spark.sql.SQLContext

Do I need to add any dependencies in pom.xml?

Spark version 1.5.2

package com.Spark.ConnectToHadoop

import org.apache.spark.SparkConf
import org.apache.spark.SparkConf
import org.apache.spark._
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.SQLContext
import org.apache.spark.rdd.RDD
//import groovy.sql.Sql.CreateStatementCommand

//import org.apache.spark.SparkConf


object CountWords  {

  def main(args:Array[String]){

    val objConf = new SparkConf().setAppName("Spark Connection").setMaster("spark://IP:7077")
    var sc = new SparkContext(objConf)
val objHiveContext = new HiveContext(sc)
objHiveContext.sql("USE test")
var rdd= objHiveContext.sql("select * from Table1")
val options=Map("path" -> "hdfs://URL/apps/hive/warehouse/test.db/TableName")
//val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   val sqlContext = new SQLContext(sc)
    import sqlContext.implicits._      //Error
val dataframe = rdd.toDF()
dataframe.write.format("orc").options(options).mode(SaveMode.Overwrite).saveAsTable("TableName")      
  }
}

My pom.xml file is as follows

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.Sudhir.Maven1</groupId>
  <artifactId>SparkDemo</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>SparkDemo</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.5.2</version>
    </dependency> 
    <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.0.0</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.10</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>0.9.1</version>
    </dependency>

     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.10</artifactId>
        <version>1.2.1</version>
    </dependency>
  <dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.2.1</version>
</dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>     

  </dependencies>
</project>

Aniket Kulkarni · Accepted Answer · 2017-07-06 14:13:22Z

7

first create

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

now we have sqlContext w.r.t sc (this will be available automatically when we launch spark-shell) now,

import sqlContext.implicits._

edited Jul 6, 2017 at 14:13

Aniket Kulkarni

13k10 gold badges70 silver badges94 bronze badges

answered May 3, 2017 at 13:34

kasi viswanath

711 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

thewaywewere Over a year ago

Welcome to SO! Your reply looks like a comment rather than a answer. Once you have sufficient reputation you will be able to comment on any post. Also check this what can I do instead. If you intended to answer, read this how-to-answer for providing quality answer.

Marsellus Wallace · Accepted Answer · 2018-03-10 21:31:01Z

6

With the release of Spark 2.0.0 (July 26, 2016) one should now use the following:

import spark.implicits._  // spark = SparkSession.builder().getOrCreate()

https://databricks.com/blog/2016/08/15/how-to-use-sparksession-in-apache-spark-2-0.html

answered Mar 10, 2018 at 21:31

Marsellus Wallace

18.7k28 gold badges96 silver badges160 bronze badges

Comments

TheMP · Accepted Answer · 2016-01-18 12:09:19Z

3

You use an old version of Spark-SQL. Change it to:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.5.2</version>
</dependency>

answered Jan 18, 2016 at 12:09

TheMP

8,4479 gold badges46 silver badges74 bronze badges

5 Comments

Alexey Romanov Over a year ago

Or, better, add a property spark.version, so when switching to a new Spark version, you only need to change it in one place.

sudhir Over a year ago

Thanks, I resolved that, but stuck with below exception : I even added dependency with spark-catalyst_2.10 : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/OverrideFunctionRegistry

TheMP Over a year ago

Are you sure that all of your spark deps are version 1.5.2? Perhaps you really should do as @Alexey Romanov suggests and introduce a spark.version variable in your pom.

sudhir Over a year ago

Ya don't know exactly , how to check that? Earlier we had spark version 1.4.1 and it was fine, recently it was upgraded to 1.5.2 added in properties too <spark.version>1.5.2</spark.version>

TheMP Over a year ago

I se that you already created a new question - stackoverflow.com/questions/34871015/… . Please paste your pom there and we can try to sort this out.

Gayan Kavirathne · Accepted Answer · 2020-08-11 19:42:11Z

1

For someone using sbt to build, update the library versions to

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.12" % "2.4.6" % "provided",
  "org.apache.spark" % "spark-sql_2.12" % "2.4.6" % "provided"
)

And then import SqlImplicits as below.

val spark = SparkSession.builder()
      .appName("appName")
      .getOrCreate()

    import spark.sqlContext.implicits._;

answered Aug 11, 2020 at 19:42

Gayan Kavirathne

3,3073 gold badges22 silver badges26 bronze badges

Comments

Arun Goudar · Accepted Answer · 2018-07-09 07:37:26Z

0

You can also use

<properties>
   <spark.version>2.2.0</spark.version>
</properties>


<dependency>
    <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
 </dependency>
  <dependency>   
      <groupId>org.apache.spark</groupId>
       <artifactId>spark-sql_2.11</artifactId>
       <version>${spark.version}</version>
  </dependency>

answered Jul 9, 2018 at 7:37

Arun Goudar

3813 silver badges6 bronze badges

Collectives™ on Stack Overflow

Spark Scala : Unable to import sqlContext.implicits._

5 Answers 5

1 Comment

Comments

5 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

Comments

5 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related