Relatively simple question I feel. Attempting to convert an integer column into epoch time (MM/DD/YYY)?
e.g., convert 881250949 --> 12/04/1997
Any advice?
Add a comment
|
1 Answer
Using from_unixtime and date_format function, we can achieve the required result:
SPARK_SCALA
val spark = SparkSession.builder().master("local[*]").getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
spark.sparkContext.setLogLevel("ERROR")
// Sample dataframe
val df = Seq(881250949).toDF("col")
df.withColumn("col", date_format(from_unixtime('col), "MM/dd/yyyy"))
.show(false)
+----------+
|col |
+----------+
|12/04/1997|
+----------+
PYSPARK
from pyspark.sql import *
from pyspark.sql.functions import *
spark = SparkSession.builder.master("local").getOrCreate()
# Sample dataframe
df = spark.createDataFrame([(1,881250949)], "id int, date int")
df.withColumn("date", date_format(from_unixtime("date"), "MM/dd/yyyy"))\
.show()
/*
+---+----------+
| id| date|
+---+----------+
| 1|12/04/1997|
+---+----------+
*/
5 Comments
Ibra22
Thanks for the prompt response Mohana. I think there is something wrong with the syntax in the response you provided. Believe there should be an ' after col
Mohana B C
Sorry didn't check on which language you asked the code. In scala that syntax works. In python you should wrap column name with in single/double quotes.
Ibra22
I keep getting an error that 'from_unixtime' is not defined. Any ideas how to explicitly define it?
Mohana B C
Edited the post, added pyspark code. Please have a look.
Ibra22
Amazing! Thank you so much. I realize that I simply had to do the following: import pyspark.sql.functions as F F.date_format(F.from_unixtime("date"), "MM/dd/yyyy")).show()