How to convert datetime from string format into datetime format in pyspark?

Question

I created a dataframe using sqlContext and I have a problem with the datetime format as it is identified as string.

df2 = sqlContext.createDataFrame(i[1])
df2.show
df2.printSchema()

Result:

2016-07-05T17:42:55.238544+0900
2016-07-05T17:17:38.842567+0900
2016-06-16T19:54:09.546626+0900
2016-07-05T17:27:29.227750+0900
2016-07-05T18:44:12.319332+0900

string (nullable = true)

Since the datetime schema is a string, I want to change it to datetime format as follows:

df3 =  df2.withColumn('_1', df2['_1'].cast(datetime()))

Here I got an error: TypeError: Required argument 'year' (pos 1) not found

What should I do to solve this problem?

Himaprasoon · Accepted Answer · 2016-08-29 06:03:48Z

4

Try this:

from pyspark.sql.types import DateType
ndf = df2.withColumn('_1', df2['_1'].cast(DateType()))

answered Aug 29, 2016 at 6:03

Himaprasoon

2,6993 gold badges28 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to convert datetime from string format into datetime format in pyspark?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related