Scala/Spark can't match function

Question

I'm trying to run the following command:

df = df.withColumn("DATATmp", to_date($"DATA", "yyyyMMdd"))

And getting this error:

<console>:34: error: too many arguments for method to_date: (e: org.apache.spark.sql.Column)org.apache.spark.sql.Column

How could I specify the exactly function to import? Has another way to avoid this error?

EDIT: Spark version 2.1

Thats strange. Can you share sample input data so that we can try to. That error should not have occured at all — Anahcolus
– Anahcolus, Commented Mar 3, 2018 at 3:18
As Andrey specified only works for 2.2, my spark is 2.1, as a "recent" version I did not expect to have version issues and thought was a spark issue. — Jader Martins
– Jader Martins, Commented Mar 3, 2018 at 18:10

Andrey Tyukin · Accepted Answer · 2018-03-02 19:05:55Z

5

As can be seen in the detailed scaladoc, the to_date function with two parameters has been added in 2.2.0, whereas the one-argument version existed since 1.5.

If you are working with an older Spark version, either upgrade, or don't use this function.

answered Mar 2, 2018 at 19:05

Andrey Tyukin

45.1k4 gold badges65 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala/Spark can't match function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related