From Array[Any] to Array[Double] in scala spark

Question

For this code :

var p = result.select("finalFeatures").head.toSeq.toArray

The results looks like this :

p: Array[Any] = Array([3.0,6.0,-0.7876947819954485,-0.21757635218517163,0.9731844373162398,-0.6641741696340382,-0.6860072219935377,-0.2990737363481845,-0.7075863760365155,0.8188108975549018,-0.8468559840943759,-0.04349947247406488,-0.45236764452589984,1.0333959313820456,0.609756607087835,-0.7106619551471779,-0.7750330808435969,-0.08097610412658443,-0.45338437108038904,-0.2952869863393396,-0.30959772365257004,0.6988768123463287,0.17049117199049213,3.2674649019757385,-0.8333373234944124,1.8462942520757128,-0.49441222531240125,-0.44187299748074166,-0.300810826687287])

I need this to be Array[Double] How would i do this ?

result is dataframe?

Apurba Pandey
– Apurba Pandey

2019-04-03 12:16:39 +00:00
Commented Apr 3, 2019 at 12:16 — Apurba Pandey
– Apurba Pandey, Commented Apr 3, 2019 at 12:16
Did you try .asInstanceOf[Array[Double]] ?

grnc
– grnc

2019-04-03 14:32:40 +00:00
Commented Apr 3, 2019 at 14:32 — grnc
– grnc, Commented Apr 3, 2019 at 14:32

SCouto · Accepted Answer · 2019-04-03 12:22:30Z

2

You can convert an Array of Any to Double as follows:

 val pAsDouble = p.map(_.toString.toDouble)

answered Apr 3, 2019 at 12:22

SCouto

7,9465 gold badges37 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

abiratsis · Accepted Answer · 2019-04-03 14:24:43Z

1

Assuming you have the following data:

val df = Seq(Array("2.3", "2.0", "5")).toDF("finalFeatures")
df.show

The output for the previous command will be:

+-------------+
|finalFeatures|
+-------------+
|[2.3, 2.0, 5]|
+-------------+

df.schema will print org.apache.spark.sql.types.StructType = StructType(StructField(finalFeatures,ArrayType(StringType,true),true)) to cast the column into double array you can do:

val doubleSeq = df.select($"finalFeatures".cast("array<double>")).head.get(0).asInstanceOf[Seq[Double]]

And doubleSeq.foreach(println _) should have the next output:

2.3
2.0
5.0

answered Apr 3, 2019 at 14:24

abiratsis

7,3414 gold badges31 silver badges49 bronze badges

6 Comments

Leothorn Over a year ago

print schema gives this root |-- finalFeatures: vector (nullable = true)

Leothorn Over a year ago

gives error org.apache.spark.sql.AnalysisException: cannot resolve 'finalFeatures' due to data type mismatch: cannot cast vector to array<double>;;

abiratsis Over a year ago

df.select($"finalFeatures".cast("array<double>")).printSchema should print root |-- finalFeatures: array (nullable = true) | |-- element: double (containsNull = true)

abiratsis Over a year ago

hi @Leothorn could post the schema of result dataframe as well?

Leothorn Over a year ago

hi @Alexandros . No resultant df . org.apache.spark.sql.AnalysisException: cannot resolve 'finalFeatures' due to data type mismatch: cannot cast vector to array<double>;; 'Project [unresolvedalias(cast(finalFeatures#9911 as array<double>), None)]

|

Collectives™ on Stack Overflow

From Array[Any] to Array[Double] in scala spark

2 Answers 2

Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related