Convert date time to unix_timestamp Scala

Question

I need to get the minimum value from the Spark data frame and transform it. Currently, I just get this value and transform it using DateTime, however, I need it in the unix_timestamp format as the result. So how can I convert DateTime to unix_timestamp either using Scala functions or Spark functions?

Here is my current code which for now returns DateTime:

val minHour = new DateTime(df.agg(min($"event_ts"))
          .as[Timestamp].collect().head))
          .minusDays(5)
          .withTimeAtStartOfDay())

I tried using Spark functions as well but I was not able to switch timestamp to start time of day (which can be achieved using DateTime withTimeAtStartOfDay function):

val minHour = new DateTime(df.agg(min($"event_ts").alias("min_ts"))
.select(unix_timestamp(date_sub($"min_ts", 5)))
.as[Long].collect().head)

mck · Accepted Answer · 2020-12-10 12:55:23Z

1

date_sub will cast your timestamp to a date, so the time will be automatically shifted to the start of day.

df.show
+-------------------+----------+
|           event_ts|event_hour|
+-------------------+----------+
|2017-05-01 00:22:01|1493598121|
|2017-05-01 00:22:08|1493598128|
|2017-05-01 00:22:01|1493598121|
|2017-05-01 00:22:06|1493598126|
+-------------------+----------+

df.agg(
    min($"event_ts").alias("min_ts")
).select(
    unix_timestamp(date_sub($"min_ts", 5)).alias("min_ts_unix")
).withColumn(
    "min_ts", $"min_ts_unix".cast("timestamp")
).show
+-----------+-------------------+
|min_ts_unix|             min_ts|
+-----------+-------------------+
| 1493164800|2017-04-26 00:00:00|
+-----------+-------------------+

answered Dec 10, 2020 at 12:55

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert date time to unix_timestamp Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related