How to cast Date column from string to datetime in pyspark/python?

Question

I have a date column with string datatype when inferred in pyspark:

Mon Oct 17 15:57:48 EST 2022

How to cast string datatype as datetime?

samkart · Accepted Answer · 2022-10-25 14:23:59Z

1

you can use the required datetime formatters - 'E MMM dd HH:mm:ss z yyyy'. the resulting timestamp will be in UTC and, thus, you'll see that it will add 5 hours to the source ts.

spark.conf.set('spark.sql.legacy.timeParserPolicy', 'LEGACY')

spark.sparkContext.parallelize([('Mon Oct 17 15:57:48 EST 2022', )]).toDF(['dt_str']). \
    withColumn('dt', func.to_timestamp('dt_str', 'E MMM dd HH:mm:ss z yyyy')). \
    show(truncate=False)

# +----------------------------+-------------------+
# |dt_str                      |dt                 |
# +----------------------------+-------------------+
# |Mon Oct 17 15:57:48 EST 2022|2022-10-17 20:57:48|
# +----------------------------+-------------------+

answered Oct 25, 2022 at 14:23

samkart

6,7133 gold badges19 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Anos Over a year ago

Thanks for your input!. I'm trying to convert to datetime instead of timestamp. Is it possible? Because when I try to convert, it is showing as not supported.

samkart Over a year ago

@Anos - datetime means timestamp. do you mean just the date part? use the to_date function instead of to_timestamp.

Anos Over a year ago

Sorry for the confusion. I thought we have separate datetime datatype conversion. So, I get to know datetime ideally means timestamp datatype. Thanks!

Anos Over a year ago

When I try to convert this Wed Oct 19 00:15:13 EST 2022, the resulting timestamp is 2022-10-19 10:45:13. Not really getting what is happening. It is not adding +5

samkart Over a year ago

@Anos - it must be because your default tz is set to be something other than UTC, and spark is adding the offset based on that tz. see this to reset your session's tz

Collectives™ on Stack Overflow

How to cast Date column from string to datetime in pyspark/python?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related