1

I am working with Spark-shell using Mongo-spark-connector to read/write data into MongoDB, while I am facing the below error, besides placing the required JARS as follows, can someone find what the problem is and help me out!

Thank you in advance

Jars:

mongodb-driver-3.4.2.jar;
mongodb-driver-sync-3.11.0.jar;
mongodb-driver-core-3.4.2.jar; 
mongo-java-driver-3.4.2.jar;
mongo-spark-connector_2.11-2.2.0.jar;
mongo-spark-connector_2.11-2.2.7.jar

Error:

scala> MongoSpark.save(dfRestaurants.write.option("spark.mongodb.output.uri", "mongodb://username:password@server_name").option("spark.mongodb.output.database", "admin").option("spark.mongodb.output.collection", "myCollection").mode("overwrite"));

**java.lang.NoClassDefFoundError: com/mongodb/MongoDriverInformation**
at com.mongodb.spark.connection.DefaultMongoClientFactory.mongoDriverInformation$lzycompute(DefaultMongoClientFactory.scala:40)
  at com.mongodb.spark.connection.DefaultMongoClientFactory.mongoDriverInformation(DefaultMongoClientFactory.scala:40)
  at com.mongodb.spark.connection.DefaultMongoClientFactory.create(DefaultMongoClientFactory.scala:49)
  at com.mongodb.spark.connection.MongoClientCache.acquire(MongoClientCache.scala:55)
  at com.mongodb.spark.MongoConnector.acquireClient(MongoConnector.scala:242)
  at com.mongodb.spark.MongoConnector.withMongoClientDo(MongoConnector.scala:155)
  at com.mongodb.spark.MongoConnector.withDatabaseDo(MongoConnector.scala:174)
  at com.mongodb.spark.MongoConnector.withCollectionDo(MongoConnector.scala:187)
  at com.mongodb.spark.sql.DefaultSource.createRelation(DefaultSource.scala:72)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)
  at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)
  at com.mongodb.spark.MongoSpark$.save(MongoSpark.scala:192)
  ... 59 elided

1 Answer 1

0

This is a typical problem when you have incorrect dependencies. In your case:

  1. Mongo Spark Connector 2.2.7 was built with driver 3.10/3.11, so it could be incompatible with driver 3.4
  2. you have 2 different versions of Mongo Spark Connector - 2.2.0 & 2.2.7 - this also could lead to problems

The better solution is to pass Maven coordinates in --packages option when starting spark shell, and allow Spark pull the package with all necessary & correct dependencies:

spark-shell --packages org.mongodb.spark:mongo-spark-connector_2.11:<version>

please make sure that you're using the Scala version that is matching your Spark version (2.12 for Spark 3.0, 2.11 for previous versions). See documentation for more details.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.