6

Hey all I have a simple 2 node cluster 1 master and 1 slave(hadoop and spark). Everything was working fine but for some reason now when I launch the master spark I can't connect to the master web ui(port 8081 i set it explicitly in the spark-env.sh file). 

Spark Command: /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/hive/lib/*:/usr/local/spark/conf/:/usr/local/spark/assembly/target/scala-2.11/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host ip-172-31-18-164 --port 7077 --webui-port 8081
========================================
16/07/23 00:47:55 INFO master.Master: Started daemon with process name: 2733@ip-172-31-18-164
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for TERM
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for HUP
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for INT
16/07/23 00:47:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls groups to: 
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls groups to: 
16/07/23 00:47:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ubuntu); groups with view permissions: Set(); users  with modify permissions: Set(ubuntu); groups with modify permissions: Set()
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/07/23 00:47:56 INFO master.Master: Starting Spark master at spark://ip-172-31-18-164:7077
16/07/23 00:47:56 INFO master.Master: Running Spark version 2.1.0-SNAPSHOT
16/07/23 00:47:56 INFO util.log: Logging initialized @1447ms
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3da770b4{/app,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@772f90b9{/app/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a2abead{/,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@34b29d61{/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@32166993{/static,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@39b78d30{/app/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2677937d{/driver/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started ServerConnector@382a571a{HTTP/1.1}{0.0.0.0:8081}
16/07/23 00:47:56 INFO server.Server: Started @1558ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'MasterUI' on port 8081.
16/07/23 00:47:56 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://172.31.18.164:8081
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@694e9b2b{/,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started ServerConnector@2f0d3a46{HTTP/1.1}{ip-172-31-18-164:6066}
16/07/23 00:47:56 INFO server.Server: Started @1587ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service on port 6066.
16/07/23 00:47:56 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73d0aa13{/metrics/master/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@359a6ac9{/metrics/applications/json,null,AVAILABLE}
16/07/23 00:47:56 INFO master.Master: I have been elected leader! New state: ALIVE
16/07/23 00:47:57 WARN server.HttpChannel: /
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
    at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
    at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
    at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.spark_project.jetty.server.Server.handle(Server.java:499)
    at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: 
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getStatus()I
    at org.spark_project.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:112)
    at org.spark_project.jetty.server.Response.sendError(Response.java:597)
    at org.spark_project.jetty.server.HttpChannel.handleException(HttpChannel.java:487)
    at org.spark_project.jetty.server.HttpConnection$HttpChannelOverHttp.handleException(HttpConnection.java:594)
    at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:387)
    at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: Unexpected thread death: org.spark_project.jetty.util.thread.QueuedThreadPool$3@15c8cd8 in MasterUI{STARTED,8<=8<=200,i=5,q=0}
16/07/23 00:47:58 INFO master.Master: Registering worker 172.31.7.238:60863 with 2 cores, 6.0 GB RAM
Improve this question

2 Answers 2

Reset to default
11

For anyone who has wandered in here from the future and can't stand esoteric runtime failures like these, here's the solution for an sbt-based Scala project.

In my case, along with spark-core and spark-hive, I had a dependency on hadoop-aws in order to correctly configure credential discovery. hadoop-aws (or something it depends on) wants javax.servlet 2.x, while spark-core likes 3.0. Putting anything mentioning servlet to the torch fixes it:

libraryDependencies ++= Seq(
...
  "org.apache.hadoop" % "hadoop-aws" % "2.8.1"
    exclude("javax.servlet",     "servlet-api")
    exclude("javax.servlet.jsp", "jsp-api")
    exclude("org.mortbay.jetty", "servlet-api"),
...
)
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, I ran into the same issue and it helps a lot. Imo, it's useless to exclude org.mortbay.jetty as it's a shaded dependency.
0

Well your stacktrace contains the error.

> 16/07/23 00:47:57 WARN server.HttpChannel: /
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
>     at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
>     at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>     at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>     at org.spark_project.jetty.server.Server.handle(Server.java:499)
>     at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
>     at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>     at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
>     at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>     at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>     at java.lang.Thread.run(Thread.java:745)

A quick google search for that error returns multiple questions for that error, i.e. SO 1 SO 2 Which Version of the servlet is in your classpath? You should have at least 3.0 since isAsyncStarted() was added then.

Improve this answer

1 Comment

yeah I realized that I was doing a * in my SPARK_CLASSPATH in the Hive Lib to get the mysql.jar But then it was bringing in another version of the servlet in. I made it explicitly call the mysql.jar file now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.