Spark org.apache.spark.sql.catalyst.analysis.UnresolvedException error in loading Hive table

Question

While trying to load data from a dataset into Hive table getting the error:

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'ipl_appl_signed_date

My dataset contains same columns as the Hive table and the column for which am getting the error has Date datatype in my code(Java) as well as in Hive.

java code:

Date IPL_APPL_SIGNED_DATE   =rs.getDate("DTL.IPL_APPL_SIGNED_DATE"); //using jdbc to get record.

Encoder<DimPolicy> encoder = Encoders.bean(Foo.class);

Dataset<DimPolicy> test=spark.createDataset(allRows,encoder); //spark is the spark session

test.write().mode("append").insertInto("someSchema.someTable"); //

I ran into a similar issue and am trying to figure out what might be the root cause. Do you use IPL_APPL_SIGNED_DATE anywhere in your query? I don't see it in the snipper, but the exception clearly says 'ipl_appl_signed_date. I think it's about Date columns. — Jacek Laskowski
– Jacek Laskowski, Commented Jan 28, 2019 at 15:13

Jacek Laskowski · Accepted Answer · 2019-01-30 13:09:10Z

1

I think the issue is due to a bug in Spark i.e. [SPARK-26379] Use dummy TimeZoneId for CurrentTimestamp to avoid UnresolvedException in CurrentBatchTimestamp, that got fixed in 2.3.3, 2.4.1, 3.0.0.

A solution is to downgrade to the version of Spark that is unaffected by the bug (or wait for a new version).

answered Jan 30, 2019 at 13:09

Jacek Laskowski

75k28 gold badges253 silver badges440 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark org.apache.spark.sql.catalyst.analysis.UnresolvedException error in loading Hive table

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related