0

I am getting this error while running spring boot application.

When I am giving build it is successful but when calling the API getting an error like below.

Please help me.

I am calling a RestController Method through API which was creating a spark session in that method.

Spark UI is getting started but in console seeing below error and also return message is not showing in the Browser.

Whitelabel Error Page

This application has no explicit mapping for /error, so you are seeing this as a fallback.

Tue Nov 07 14:36:51 IST 2017 There was an unexpected error (type=Internal Server Error, status=500). com/fasterxml/jackson/databind/Module

    java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module
        at java.net.URLClassLoader.findClass(Unknown Source) ~[na:1.8.0_131]
        at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_131]
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) ~[na:1.8.0_131]
        at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_131]
        at java.lang.Class.forName0(Native Method) ~[na:1.8.0_131]
        at java.lang.Class.forName(Unknown Source) ~[na:1.8.0_131]
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) ~[na:na]
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) ~[na:na]
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) ~[na:na]
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) ~[na:na]
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) ~[na:na]
        at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:194) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:102) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:522) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) ~[spark-core_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) ~[spark-sql_2.11-2.1.0.jar:2.1.0]
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) ~[spark-sql_2.11-2.1.0.jar:2.1.0]
        at scala.Option.getOrElse(Option.scala:121) ~[na:na]
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) ~[spark-sql_2.11-2.1.0.jar:2.1.0]
        at com.tam.ui.DemoController.home(DemoController.scala:19) ~[classes/:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_131]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.12.RELEASE.jar:4.3.12.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) ~[tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_131]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar:8.5.23]
        at java.lang.Thread.run(Unknown Source) [na:1.8.0_131]



My Maven pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>TestSpringBoot</groupId>
  <artifactId>TestSpringBoot</artifactId>
  <version>0.0.1-SNAPSHOT</version>

   <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.8.RELEASE</version>

    </parent>

     <properties>

        <java.version>1.8</java.version>
        <!--<start-class>me.wonwoo.Application</start-class>-->
        <scala.version>2.11.8</scala.version>
     </properties>


     <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
            <exclusion>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>

                  </exclusion>
                  <exclusion>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
                  </exclusion>
                  <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
                  </exclusion>
                  <exclusion>
                  <groupId>com.fasterxml.jackson.module</groupId>
                    <artifactId>jackson-module-scala_2.11</artifactId>
                  </exclusion>
                   <exclusion>
                  <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-parameter-names</artifactId>

                  </exclusion>

            </exclusions>     
        </dependency>


        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.0</version>
        <scope>provided</scope> 
            <exclusions>
            <exclusion>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>

                  </exclusion>
                  <exclusion>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
                  </exclusion>
                  <exclusion>
                  <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-parameter-names</artifactId>

                  </exclusion>
                  <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
                  </exclusion>
                  <exclusion>
                  <groupId>com.fasterxml.jackson.module</groupId>
                    <artifactId>jackson-module-scala_2.11</artifactId>
                  </exclusion>
            </exclusions>     
        </dependency>


        <dependency>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
             <version>2.9.1</version>
              <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
        </exclusion>
    </exclusions>
                  </dependency>
                  <dependency>
                 <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
             <version>2.9.1</version>
                  </dependency>
                  <dependency>
                <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
               <version>2.9.1</version>
                  </dependency>
                  <dependency>
                 <groupId>com.fasterxml.jackson.module</groupId>
                 <artifactId>jackson-module-scala_2.11</artifactId>
             <version>2.9.1</version>
            </dependency>

            <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-parameter-names -->
            <dependency>
                <groupId>com.fasterxml.jackson.module</groupId>
                <artifactId>jackson-module-parameter-names</artifactId>
                <version>2.9.2</version>
            </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.1.0</version>
        <scope>provided</scope>           
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.1.1</version>
    <scope>provided</scope>
</dependency>

<dependency>

   <groupId>org.springframework.boot</groupId>

   <artifactId>spring-boot-starter-validation</artifactId>

</dependency>

        </dependencies>
     <build>
     <plugins>
     <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>


     </plugins>

     </build>
</project>
Improve this question

2 Answers 2

Reset to default
2

What you have is multiple versions of jackson in your uber jar,
so what you need is shading rules to rename the the jackson libs that spark uses to some other name. you will have both versions in the classpath but under different names. I myself use sbt so I've never done it with maven but you can start here maven-shade-plugin it also explains what shading is.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
But in a nutshell, what shading does is:
Rename all usages of some lib com.fasterxml.jackson. **
to some rule you define, for example shaded_xml.@1.
and what you get in result is:

  1. another directory in the class path with the name: shaded_xml, and in it, all your classes as in the original com.fasterxml.jackson
  2. all the classes using com.fasterxml.jackson. ** rewritten to use shaded_xml. ** instead.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To check your shading rules, you can actually open your jar and look for the new directory in the class path.
You can also look in some class file you know has usage of the original lib name and check if it was renamed

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Remove Jackson related libs from exclusions and it should work.

Improve this answer

1 Comment

spark uses jackson as it's json serilizer, parser, etc... so removing it completely will not solve the problem but will make things worse, this means he has different versions of jackson so he needs to add shading rules which will give different named to different versions and all needed versions could be found in the classpath, but again - under different names.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.