Why does Spark fail with "value write is not a member of org.apache.spark.sql.DataFrameReader [error]"?

Question

I have two almost identical write into db . scala statement ,however one trowing me an error the other not and i don't understand how to fix it ? any ideas ?

this statement is working

df_pm_visits_by_site_trn.write.format("jdbc").option("url", db_url_2).option("dbtable", "pm_visits_by_site_trn").option("user", db_user).option("password", db_pwd).option("truncate","true").mode("overwrite").save()

this one doesn't work and throwing me compiling error

df_trsnss .write.format("jdbc").option("url", db_url_2).option("dbtable", "df_trsnss")               .option("user", db_user).option("password", db_pwd).option("truncate","true").mode("overwrite").save()

_dev.scala:464: value write is not a member of org.apache.spark.sql.DataFrameReader [error] df_trsnss.write.format("jdbc").option("url", db_url_2).option("dbtable", "trsnss").option("user", db_user).option("password", db_pwd).option("truncate","true").mode("overwrite").save()

if i delete my second write statement or just simply comment it out whole code is compiling with no errors.

Silvio · Accepted Answer · 2018-10-14 21:25:50Z

6

Based on the error message, df_trsnss is a DataFrameReader, not a DataFrame. You likely forgot to call load.

val df_trsnss = spark.read.format("csv")

instead of

val df_trsnss = spark.read.format("csv").load("...")

edited Oct 14, 2018 at 21:25

answered Oct 14, 2018 at 20:37

Silvio

4,22726 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why does Spark fail with "value write is not a member of org.apache.spark.sql.DataFrameReader [error]"?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related