what is the difference between array<string> and Array<String> in Spark?

Question

I used this code to aggragate the grouped data:

val result=union_df.orderBy(desc("timestamp")).groupBy("id").agg(collect_set("region") as "region")

Then I got the datatype:

org.apache.spark.sql.DataFrame = [id: string, region: array<string>]

What is the different between array<string> and Array<String>? How do I iterate over array<string> in map function (there is no getArray function for Row)?

array<string> and string are not Scala types, they are just results of toString call on a DataType which is called by toString on DataFrame. — Alexey Romanov
– Alexey Romanov, Commented May 24, 2017 at 7:16

koiralo · Accepted Answer · 2017-05-24 03:27:02Z

2

There is a getSeq() function that returns a array or you can use getAs method by getAs[Array[String]]() or getAs[Seq[String]]()

array<string> is scala.collection.mutable.WrappedArray[String]] which is same as a Array<String> with a wrapper on it which is used inside a Row.

edited May 24, 2017 at 3:27

answered May 24, 2017 at 3:17

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

what is the difference between array<string> and Array<String> in Spark?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related