Spark - Difference between array() and Array()

Question

I was converting in the Spark Shell (1.6) a List of strings into an array like this:

val mapData = List("column1", "column2", "column3")
val values = array(mapData.map(col): _*)

The type of values is:

values: org.apache.spark.sql.Column = array(column1,column2,column3)

Everything fine, but when I start developing in Eclipse I got the error:

not found: value array

So I changed to this:

val values = Array(mapData.map(col): _*)

The problem I faced then was that the type of value now changed and the udf which was consuming it doesn't accept this new type:

values: Array[org.apache.spark.sql.Column] = Array(column1, column2, column3)

Why I am not able to use array() in my IDE as in the Shell (what import am I missing)? and why array produce a org.apache.spark.sql.Column without the Array[] wrapper?

Edit: The udf function:

def replaceFirstMapOfArray = 
udf((p: Seq[Map[String, String]], o: Seq[Map[String, String]]) =>
{
    if((null != o && null !=p)){
        if ( o.size == 1 ) p
        else p ++ o.drop(1)
    }else{
        o
    }
})

a udf function would require a column argument to be passed and values in a primitive array . Thats why the error is happening. Shankar's answer below should have answered your confusion — Anahcolus
– Anahcolus, Commented Mar 21, 2018 at 7:30

philantrovert · Accepted Answer · 2018-03-21 08:28:01Z

3

val mapData = List("column1", "column2", "column3")
val values = array(mapData.map(col): _*)

Here, Array or List is the collection of objects

where as array in array(mapData.map(col): _*) is a spark function that creates a new column with type array for the same datatype columns.

For this to be used you need to import

import org.apache.spark.sql.functions.array

You can see here about the array

/**    
 * Creates a new array column. The input columns must all have the same data type.  
 * @group normal_funcs    
 * @since 1.4.0   
 */

 @scala.annotation.varargs
 def array(cols: Column*): Column = withExpr { 
   CreateArray(cols.map(_.expr)) 
 }

edited Mar 21, 2018 at 8:28

philantrovert

10.1k3 gold badges43 silver badges65 bronze badges

answered Mar 21, 2018 at 7:14

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ignacio Alorre Over a year ago

Thanks. So how could I modify me code so Eclipse doesn't complain?

koiralo Over a year ago

did you import import org.apache.spark.sql.functions.array

Ignacio Alorre Over a year ago

Yes was that import. Thanks

Collectives™ on Stack Overflow

Spark - Difference between array() and Array()

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related