1

When I run my code in the local code, it works fine. However, when I run it in the cluster, it seems that some dependency is missed in my Jar file:

18/05/23 10:46:02 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)

This is SBT file:

scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"
val mahoutVersion = "0.13.1"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "org.sedis" %% "sedis" % "1.2.2",
  "org.scalactic" %% "scalactic" % "3.0.0",
  "org.scalatest" %% "scalatest" % "3.0.0" % "test",
  "com.github.nscala-time" %% "nscala-time" % "2.14.0",
  "com.github.scopt" %% "scopt" % "3.3.0"
)
unmanagedSourceDirectories in Compile += baseDirectory.value / "lib"
resolvers += "typesafe repo" at " http://repo.typesafe.com/typesafe/releases/"
resolvers += Resolver.mavenLocal
assemblyMergeStrategy in assembly := {
  case "plugin.properties" => MergeStrategy.discard
  //case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case PathList("org", "joda", "time", "base", "BaseDateTime.class") => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith "package-info.class" => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "WrapDynaClass.class") => MergeStrategy.first
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "aop", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "intercept", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
  case PathList("org", "apache", "commons", "collections", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("assets", "org", "apache", "commons", "math3", "exception", "util", "LocalizedFormats_fr.properties") => MergeStrategy.first
  case PathList("com", "google", "common", "annotations", "Beta.class") => MergeStrategy.first
  case PathList("com", "google", "common", "base", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "cache", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "collect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "eventbus", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "hash", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "io", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "math", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "net", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "primitives", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "reflect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "util", "concurrent", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "differentiation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "function", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "integration", "gauss", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "interpolation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "polynomials", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "solvers", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "complex", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "dfp", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "distribution", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "filter", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fraction", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "genetics", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "oned", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "threed", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "twod", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "partitioning", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "distance", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "events", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "nonstiff", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "sampling", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "gradient", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "noderiv", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "vector", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "direct", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "general", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "primes", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "random", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "special", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "correlation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "moment", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "summary", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "inference", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "ranking", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "regression", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "transform", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "logging", xs @ _*) => MergeStrategy.first
  case PathList("overview.html") => MergeStrategy.first
  case PathList("mime.types") => MergeStrategy.first
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

What is wrong in my Jar? In IntelliJ IDEA I selected the class inspection option and pasted SSLConnectionSocketFactory. This class was found and referenced httpclient-4.3.6.jar

Improve this question

1 Answer 1

Reset to default
1

Spark comes with many libraries and you probably have conflict with one of them.

I think I had a similar issue with io.netty.netty-all package. We ended up upgrading that package on the server to a slightly more recent minor release, but this was because we were building an integration for spark that was deployed on the nodes.

You can try to deploy your spark app with these parameters

--conf 'spark.driver.extraJavaOptions=-verbose:class'
--conf 'spark.executor.extraJavaOptions=-verbose:class'

those allow you to see the classpath. I'm pretty sure you will see the conflicting package there. I'd start checking what version of netty is loaded as I'm pretty sure it depends on the httpclient.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

I know that I have 2 conflicting packages, one of them is older version, another one is newer version. But how can I pick only the new version?
You can use sbt dependency overrides to define which version is packaged dependencyOverrides += "log4j" % "log4j" % "1.2.16", more information here: scala-sbt.org/1.x/docs/Library-Management.html. If you can't make both work with the same version of the dependency, then I would look at shading, this seems to explain it pretty well: yonatanwilkof.net/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.