Spark SQL - Incorrect conversion of oracle date data type to timestamp(java.sql) while importing from oracle

Question

I am using Spark SQL to import data from Oracle as below-

Class.forName("oracle.jdbc.driver.OracleDriver")  
var info : java.util.Properties = new java.util.Properties()  
info.put("user", user)  
info.put("password", password) 
val jdbcDF = spark.read.jdbc(jdbcURL, tableFullName, info)

Table Schema:

SERVICE_DATE - DATE

Spark while importing converts columns having "date" data type to "java.sql.timestamp" data type. During this conversion, I am facing an issue for dates which follows daylight saving time.

For eg:

oracle: SERVICE_DATE = 2008-03-09 02:49:00.0 [DATE]
spark: SERVICE_DATE = 2008-03-09 03:49:00.0 [TIMESTAMP]

Is this issue with the conversion of oracle date to JDBC timestamp?

Is this solved?

Vijay Anand Pandian
– Vijay Anand Pandian

2020-08-26 09:26:15 +00:00
Commented Aug 26, 2020 at 9:26 — Vijay Anand Pandian
– Vijay Anand Pandian, Commented Aug 26, 2020 at 9:26

anuragagarwal561994 · Accepted Answer · 2018-03-16 10:51:44Z

4

From https://github.com/apache/spark/pull/18411/files/aefd028883bc27cd5929e80dff29d2b15aa114b2

You can see, there is a property called oracle.jdbc.mapDateToTimestamp which is true by default. If you set it to false the Date columns won't be marked as Timestamp.

answered Mar 16, 2018 at 10:51

anuragagarwal561994

661 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

BdEngineer Over a year ago

any clue of stackoverflow.com/questions/56020103/…?

Collectives™ on Stack Overflow

Spark SQL - Incorrect conversion of oracle date data type to timestamp(java.sql) while importing from oracle

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related