create a spark dataframe column consists of a list as data type

Question

I have an existing dataframe 'df' with a column 'list_len' and I want to create a column that consists of empty strings with length indicated by the value of 'list_len'.

I tried to do df.withColumn('new_list', array(['']*col('list_len'))).show() in pyspark but it did not work.

any idea/help is greatly appreciated!

+---------+------------------+
|list_len |        new_list  |
+---------+------------------+
|        1|              ['']|
|        3|      ['', '', '']|
|        2|          ['', '']|
+----------------------------+

Dharman · Accepted Answer · 2020-10-27 16:48:58Z

1

scala:

import org.apache.spark.sql.functions.{lit,array_repeat}
import spark.implicits._

val df = Seq(1, 2, 3).toDF("list_len")
df.withColumn("new_list", array_repeat(lit(""), $"list_len"))

reference: https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html#array_repeat-org.apache.spark.sql.Column-org.apache.spark.sql.Column-

pyspark:

from pyspark.sql.functions import lit, array_repeat, col
df.withColumn("new_list", array_repeat(lit(""), col("list_len")))

reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.array_repeat

edited Oct 27, 2020 at 16:48

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Oct 27, 2020 at 16:42

Matt

6505 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

create a spark dataframe column consists of a list as data type

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related