Creating a SQLContext Dataset from an RDD containing arrays of Strings in Spark [duplicate]

Question

So I have a variable data which is a RDD[Array[String]]. I want to iterate over it and compare adjacent elements. To do this I must create a dataset from the RDD.

I try the following, sc is my SparkContext:

import org.apache.spark.sql.SQLContext

val sqc = new SQLContext(sc)
val lines = sqc.createDataset(data)

And I get the two following errors:

Error:(12, 34) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._ Support for serializing other types will be added in future releases. val lines = sqc.createDataset(data)

Error:(12, 34) not enough arguments for method createDataset: (implicit evidence$4: org.apache.spark.sql.Encoder[Array[String]])org.apache.spark.sql.Dataset[Array[String]]. Unspecified value parameter evidence$4. val lines = sqc.createDataset(data)

Sure, I understand I need to pass an Encoder argument, however, what would it be in this case and how do I import Encoders? When I try myself it says that createDataset does not take that as argument.

There are similar questions, but they do not answer how to use the encoder argument. If my RDD is a RDD[String] it works perfectly fine, however in this case it is RDD[Array[String]].

I dont consider it duplicate since I have already read through those questions. — osk
– osk, Commented Dec 4, 2017 at 8:31
So if I import it, how do I use the encoder? (What do I pass to the 2nd argument?) — osk
– osk, Commented Dec 4, 2017 at 8:31
Once the implicits are in scope, Spark will automatically convert whatever it can. If you want to be more specific about it, you can use : sqc.createDataset(rdd)(newStringArrayEncoder) — philantrovert
– philantrovert, Commented Dec 4, 2017 at 8:44
import sqlContext.implicits._ for spark ver < 2 and spark.implicits._ for Spark2+ — philantrovert
– philantrovert, Commented Dec 4, 2017 at 8:47

Anahcolus · Accepted Answer · 2017-12-07 15:57:47Z

1

All of the comments in the question are trying to tell you the following things

You say you have RDD[Array[String]], which I create by doing the following

val rdd = sc.parallelize(Seq(Array("a", "b"), Array("d", "e"), Array("g", "f"), Array("e", "r")))   //rdd: org.apache.spark.rdd.RDD[Array[String]] = ParallelCollectionRDD[0] at parallelize at worksheetTest.sc4592:13

Now converting the rdd to dataframe is to call .toDF but before that you need to import implicits._ of sqlContext as below

val sqc = new SQLContext(sc)
import sqc.implicits._
rdd.toDF().show(false)

You should have dataframe as

+------+
|value |
+------+
|[a, b]|
|[d, e]|
|[g, f]|
|[e, r]|
+------+

Isn't this all simple?

answered Dec 7, 2017 at 15:57

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Creating a SQLContext Dataset from an RDD containing arrays of Strings in Spark [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related