4

I am installing SparkR in my Windows 8.1 from this tutorial https://www.linkedin.com/pulse/setting-up-sparkr-windows-machine-ramabhadran-kapistalam. I ended it so I guess it's well implemented. The problem is when I try to run an example with a simple Data Frame:

Error in handleErrors(returnStatus, conn) : 
  java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
    at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
    at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
    at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:62)
    at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:552)
    at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:307)
    at org.apache.spark.sql.api.r.SQLUtils$.createDF(SQLUtils.scala:139)
    at org.apache.spark.sql.api.r.SQLUtils.createDF(SQLUtils.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at

This is my code in R:

 Sys.setenv(SPARK_HOME = "C:/Spark/spark-2.1.1-bin-hadoop2.7")
 .libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"), .libPaths()))
 sparkR.session(appName = "SparkR-DataFrame-example")
 df <- as.DataFrame(faithful)

And i saw a solution that I had to configure the sparkr session by adding:

sparkR.session(master = "local[*]", sparkConfig = list(spark.driver.memory = "1g", spark.sql.warehouse.dir = "file:///somelocaldirectory"))

I tried to edit the spark.sql.warehouse.dir with a data file but the error persisted

1 Answer 1

2

I had exact same problem, starting R-studio or R as administrator solved this issue.

to start as administrator, right click on R-studio or R and select run as administrator and then your commands should work just fine.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.