Pyspark column: Convert data in string format to timestamp format

Question

I am trying to convert one pyspark column in string format to timestamp format

I tried as below but I get null values for that column.

Thanks

Example data:   06/19/17 00:00:00

df = df.withColumn("date", to_timestamp(col("date"), "MM/dd/yyy HHmm"))

When I tried casting as below

df = df.withColumn("date",(col("invoice_date").cast(TimestampType())))

output: 2017-06-19T00:00:00.000+0000

Expected output:

 06/19/17 00:00:00

Surya · Accepted Answer · 2020-11-13 03:34:46Z

1

You can do:

from pyspark.sql import functions as F

df.withColumn("date", F.from_unixtime(F.unix_timestamp("date", \
    'MM/dd/yy HH:mm:ss'),'MM-dd-yy HH:mm:ss')).show()

+-----------------+
|             date|
+-----------------+
|06-19-17 00:00:00|
+-----------------+

answered Nov 13, 2020 at 3:34

Surya

3,4293 gold badges22 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark column: Convert data in string format to timestamp format

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related