44: error: value read is not a member of object org.apache.spark.sql.SQLContext

Question

I am using Spark 1.6.1, and Scala 2.10.5. I am trying to read the csv file through com.databricks. While launching the spark-shell, I use below lines as well

spark-shell --packages com.databricks:spark-csv_2.10:1.5.0 --driver-class-path path to/sqljdbc4.jar, and below is the whole code

import java.util.Properties
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext


val conf = new SparkConf().setAppName("test").setMaster("local").set("spark.driver.allowMultipleContexts", "true");
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

import sqlContext.implicits._

val df = SQLContext.read().format("com.databricks.spark.csv").option("inferScheme","true").option("header","true").load("path_to/data.csv");

I am getting below error:-

error: value read is not a member of object org.apache.spark.sql.SQLContext, and the "^" is pointing toward "SQLContext.read().format" in the error message.

I did try the suggestions available in stackoverflow, as well as other sites as well. but nothing seems to be working.

T. Gawęda · Accepted Answer · 2017-03-08 11:27:52Z

1

SQLContext means object access - static methods in class.

You should use sqlContext variable, as methods are not static, but are in class

So code should be:

val df = sqlContext.read.format("com.databricks.spark.csv").option("inferScheme","true").option("header","true").load("path_to/data.csv");

answered Mar 8, 2017 at 11:27

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3521180 Over a year ago

But now I get another error conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@b43db07 sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@774cc4a9 sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@62d42eb7 java.lang.RuntimeException: Error in configuring object What should I make out of this?

T. Gawęda Over a year ago

@user3521180 It's related to: stackoverflow.com/questions/30263646/…. It's unrelated to this question and solving will require more info - full code and exception stacktrace - so I suggest posting new question

Collectives™ on Stack Overflow

44: error: value read is not a member of object org.apache.spark.sql.SQLContext

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related