0

We are using Spark 3.3.2 with Scala 2.12.15 and JDK 11.0.16

I don't have any scala code. Purely python and RDD.

Still we are getting the below error in rancher.

java.io.InvalidClassException: org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$RetrieveSparkAppConfig$; local class incompatible: stream classdesc serialVersionUID = .....

I have searched so many topics in Stack overflow. But everywhere they are telling version incompatability. We have all versions up to date and we can't modify..

Please suggest some proper solution. Thanks in advance

Improve this question
8
  • Is org.apaches a typo? Should it be org.apache? "I don't have any scala code" Are you using PySpark? Then you do have Scala code. The class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveSparkAppConfig is from spark-core. Just in case, hardly this is the reason but did you try to run with JDK 8 rather than 11? Commented Apr 11, 2023 at 8:21
  • The question says that most probably java.io.InvalidClassException: local class incompatible: stream classdesc is because incompatible versions of a dependency are used (with different serialVersionUID). Commented Apr 11, 2023 at 8:22
  • Let's look at your Java classpath. Can you run the following Scala code inside your PySpark environment? scastie.scala-lang.org/DmytroMitin/KfLOLvdmSLqOL6rKfnKFow or at least scastie.scala-lang.org/DmytroMitin/KfLOLvdmSLqOL6rKfnKFow/1 This is how to run Scala code in PySpark: aseigneurin.github.io/2016/09/01/… stackoverflow.com/questions/36023860/… Why do you say that you can't modify the versions? It's possible that you'll have to. Commented Apr 11, 2023 at 8:22
  • "We have all versions up to date" Why do you consider Scala 2.12.15 up to date? mvnrepository.com/artifact/org.scala-lang/scala-library Commented Apr 11, 2023 at 9:40
  • 1
    Yes it's docker image. We are just using it without modifying the versions. Commented Apr 12, 2023 at 12:09
Related questions
2031
Getting the class name of an instance
0
Spark in cluster with Docker: BlockManagerId; local class incompatible
2
Unable to connect to spark on remote system
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.