How to make column from a list in scala/pyspark dataframe ? Error: The feature is not supported: literal for 'List()

Question

enter code hereI am practising to add a list into dataframe col. I can def udf and register and then apply on dataframe but I want to try a different approach that extracting a list from dataframe col and them map it and then readd to the original dataframe in new column.

val df = spark.createDataFrame(Seq(("A",1),("B",2),("C",3))).toDF("Str", "Num")
+---+---+
|Str|Num|
+---+---+
|  A|  1|
|  B|  2|
|  C|  3|
+---+---+

list collected:

scala> var ls : List[String] = df.select("Str").collect().map(f=>f.getString(0)).toList
var ls: List[String] = List(A, B, C, d)

Transformation:

def f(x : String) : String = {
  if (x=="A") {x + "100"}
  else {x + x.length.toString}
  }

apply transformation:

scala> ls.map(x => f(x))
val res95: List[String] = List(A100, B1, C1, d1)

add column from the list: ERROR

import org.apache.spark.sql.functions.{lit,col}
df.withColumn("new", lit(ls)).show()

error: org.apache.spark.SparkRuntimeException: The feature is not supported: literal for 'List(A100, B1, C1)' of class scala.collection.immutable.$colon$colon. 

//Please correct here

Thanks for that. can you please help for creating column as well — guest
– guest, Commented Sep 14, 2022 at 8:09

Emiliano Martinez · Accepted Answer · 2022-09-15 08:09:57Z

1

Create the udf:

val myUdf = udf { x: String =>
   if (x=="A") {x + "100"}
   else {x + x.length.toString}
 }

and the apply to the df:

df.withColumn("new", myUdf(col("Str")))

to add a new column from a List:

df.withColumn("fromListColumn", array(Seq("one", "two").map(lit(_)):_*))

edited Sep 15, 2022 at 8:09

answered Sep 14, 2022 at 14:53

Emiliano Martinez

4,1432 gold badges13 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to make column from a list in scala/pyspark dataframe ? Error: The feature is not supported: literal for 'List()

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related