Pass list item as input for withColumn (Pyspark)

Question

Working on a Spark Dataframe where I want to adjust the content of a field. Input for the adjustment comes from a list, however, when I pass the information in the list as arguments I get an error:

AssertionError: col should be Column

fld = ["As_Of_Date","date_format('As_Of_Date,'yyyyMMdd')"]

df.withColumn(fld[0],fld[1])

if I pass it directly without the list, it does work. What am I doing wrong and what should I change to make it work?

Vamsi Prabhala · Accepted Answer · 2020-01-13 16:52:30Z

1

You don't need to quote the function date_format here.

fld = ['As_Of_Date',date_format('As_Of_Date','yyyyMMdd')]
df = df.withColumn(fld[0],fld[1])

edited Jan 13, 2020 at 16:52

answered Jan 13, 2020 at 16:39

Vamsi Prabhala

49.4k4 gold badges41 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Vamsi Prabhala Over a year ago

check the edit..assignment to the data frame with the transformed column was missing which has been added.

Collectives™ on Stack Overflow

Pass list item as input for withColumn (Pyspark)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related