Calculate time between two dates in pyspark

Question

Hoping this is fairly elementary. I have a Spark dataframe containing a Date column, I want to add a new column with number of days since that date. Google fu is failing me.

Here's what I've tried:

from pyspark.sql.types import *
import datetime
today = datetime.date.today()

schema = StructType([StructField("foo", DateType(), True)])
l = [(datetime.date(2016,12,1),)]
df = sqlContext.createDataFrame(l, schema)
df = df.withColumn('daysBetween',today - df.foo)
df.show()

it fails with error:

u"cannot resolve '(17212 - foo)' due to data type mismatch: '(17212 - foo)' requires (numeric or calendarinterval) type, not date;"

I've tried fiddling around but gotten nowhere. I can't think that this is too hard. Can anyone help?

jamiet · Accepted Answer · 2017-02-15 22:14:15Z

12

OK, figured it out

from pyspark.sql.types import *
import pyspark.sql.functions as funcs
import datetime
today = datetime.date(2017,2,15)

schema = StructType([StructField("foo", DateType(), True)])
l = [(datetime.date(2017,2,14),)]
df = sqlContext.createDataFrame(l, schema)
df = df.withColumn('daysBetween',funcs.datediff(funcs.lit(today), df.foo))
df.collect()

returns [Row(foo=datetime.date(2017, 2, 14), daysBetween=1)]

answered Feb 15, 2017 at 22:14

jamiet

12.7k22 gold badges117 silver badges203 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Abhishek Arora · Accepted Answer · 2017-11-30 22:18:33Z

8

You can simply do the following:

import pyspark.sql.functions as F

df = df.withColumn('daysSince', F.datediff(F.current_date(), df.foo))

answered Nov 30, 2017 at 22:18

Abhishek Arora

1,0042 gold badges11 silver badges16 bronze badges

1 Comment

gabra Over a year ago

So others can know: the differences are in days spark.apache.org/docs/2.1.0/api/python/…

Collectives™ on Stack Overflow

Calculate time between two dates in pyspark

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related