spark collect as Array[T] and not as Array[Row] from data frame

Question

I can collect a column like this using the RDD API.

df.map(r => r.getAs[String]("column")).collect

However, as I am initially using a Dataset I rather would like to not switch the API level. A simple df.select("column).collect returns an Array[Row] where the .flatten operator no longer works. How can I collect to Array[T e.g. String] directly?

Have you tried df.select("column).as[String].collect? Writing from memory, if works I'll post normal answer :) — T. Gawęda
– T. Gawęda, Commented Nov 22, 2016 at 21:07

Lokesh Yadav · Accepted Answer · 2016-11-22 21:09:12Z

16

With Datasets ( Spark version >= 2.0.0 ), you just need to convert the dataframe to dataset and then collect it.

df.select("column").as[String].collect()

would return you an Array[String]

answered Nov 22, 2016 at 21:09

Lokesh Yadav

9982 gold badges9 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Georg Heiler Over a year ago

nice. That works fine. But why isn't the type inferred from the schema of the dataset automatically?

T. Gawęda Over a year ago

@GeorgHeiler yes and not :) It is inferred, but it will be Row[String], because in select you can put many column names. Converter will automatically convert to String

Collectives™ on Stack Overflow

spark collect as Array[T] and not as Array[Row] from data frame

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related