Spark access nested column

Question

I want to access a column which contains a nested array similar to Spark combine columns as nested array

columnsMap
res95: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Array[Double]]] = Map(col1 -> Map(A -> WrappedArray(0.5, 0.6666666666666666), d -> WrappedArray(0.25, 0.3333333333333333), c -> WrappedArray(0.25, 1.0)), col2 -> Map(a -> WrappedArray(0.0, 0.0), g -> WrappedArray(0.0, 0.0), B -> WrappedArray(0.5, 0.6666666666666666), c -> WrappedArray(0.25, 1.0), d -> WrappedArray(0.25, 1.0)), col3TooMany -> Map(C -> WrappedArray(0.75, 0.6), jkl -> WrappedArray(0.0, 0.0), t -> WrappedArray(0.25, 1.0)))

scala> columnsMap.get("col1").get.get("A")
res96: Option[Array[Double]] = Some(WrappedArray(0.5, 0.6666666666666666))

How can I access the WrappedArray in plain Scala?

scala> columnsMap.get("col1").get.get("A").get
java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [D
  ... 42 elided

Daniel de Paula · Accepted Answer · 2016-12-20 14:18:22Z

1

You can use getOrElse to avoid working directly with Options:

val columnsMap = Map("col1" -> Map("A" -> Seq(1.0, 2.0)))
val nestedArray = columnsMap.getOrElse("col1", Map()).getOrElse("A", Seq(0.0))

Then, to get the first element, you can do:

nestedArray.head

Or, if at collecting the rows you did not properly typed the array to Seq, you may need to do:

nestedArray.asInstanceOf[Seq[Double]].head

But the ideal should be to collect with the right type. For instance, if you used a row.getAs at some point, it should be:

row.getAs[Map[String, Map[String, Seq[Double]]]](colName)

edited Dec 20, 2016 at 14:18

answered Dec 20, 2016 at 13:47

Daniel de Paula

18k9 gold badges75 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Georg Heiler Over a year ago

Your first solution returns: Any = WrappedArray(0.5, 0.6666666666666666) How could I access the first double value?

Georg Heiler Over a year ago

As Array is of type WrappedArray e.g. no real Array it is unclear to me how to access it.

Daniel de Paula Over a year ago

@GeorgHeiler Leaving like that, you can have an empty Array, so you need to test for it. You can do nestedArray.headOption to get an Option[Double] which you can then test for emptiness. Alternatively, you can just change the second getOrElse to getOrElse("A", Array(0.0)) to make sure the array is not empty, then you can just do nestedArray.head.

Georg Heiler Over a year ago

this does not work e.g.

columnsMap.getOrElse("col1", Map()).getOrElse("A", Array(0.0)).head java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [D   ... 42 elided

Daniel de Paula Over a year ago

@GeorgHeiler Try nestedArray.toArray.head or nestedArray.toList.head

|

Collectives™ on Stack Overflow

Spark access nested column

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related