Creating a Spark DataFrame from a single string

Question

I'm trying to take a hardcoded String and turn it into a 1-row Spark DataFrame (with a single column of type StringType) such that:

String fizz = "buzz"

Would result with a DataFrame whose .show() method looks like:

+-----+
| fizz|
+-----+
| buzz|
+-----+

My best attempt thus far has been:

val rawData = List("fizz")
val df = sqlContext.sparkContext.parallelize(Seq(rawData)).toDF()

df.show()

But I get the following compiler error:

java.lang.ClassCastException: org.apache.spark.sql.types.ArrayType cannot be cast to org.apache.spark.sql.types.StructType
    at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:413)
    at org.apache.spark.sql.SQLImplicits.rddToDataFrameHolder(SQLImplicits.scala:155)

Any ideas as to where I'm going awry? Also, how do I set "buzz" as the row value for the fizz column?

Update:

Trying:

sqlContext.sparkContext.parallelize(rawData).toDF()

I get a DF that looks like:

+----+
|  _1|
+----+
|buzz|
+----+

2 revs user6022341 · Accepted Answer · 2016-10-10 17:45:24Z

9

Try:

sqlContext.sparkContext.parallelize(rawData).toDF()

In 2.0 you can:

import spark.implicits._

rawData.toDF

Optionally provide a sequence of names for toDF:

sqlContext.sparkContext.parallelize(rawData).toDF("fizz")

edited Oct 10, 2016 at 17:45

community wiki

2 revs
user6022341

Sign up to request clarification or add additional context in comments.

3 Comments

smeeb Over a year ago

Thanks @LostInOverflow (+1) - I think I'm almost there, please see my update. I am getting a single-row DF, with the correct value in it ("buzz" string), but the column name is "_1"...thoughts?

KiranM Over a year ago

Dataframe is like dataset in tabular format with column/title. In the first case, you created dataframe with no column name specified, so it assigns default columns as "_1", "_2".

Martin Wunderlich Over a year ago

How would this work in Java? sparkContext.parallelize takes two additional parameters: numSlices and ClassTag. The 2nd isn't clear to me.

Martin Wunderlich · Accepted Answer · 2021-11-18 10:48:26Z

0

In Java, the following works:

List<String> textList = Collections.singletonList("yourString");
SQLContext sqlContext = new SQLContext(sparkContext);
Dataset<Row> data = sqlContext
      .createDataset(textList, Encoders.STRING())
      .withColumnRenamed("value", "text");

answered Nov 18, 2021 at 10:48

Martin Wunderlich

1,9641 gold badge19 silver badges43 bronze badges

Collectives™ on Stack Overflow

Creating a Spark DataFrame from a single string

Update:

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Update:

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related