1

I am getting error

Exception in thread "main" java.lang.NoClassDefFoundError

Here is my complete program:

object StatefulNetworkWordCount {
  def updateFunction(newValues: Seq[Int], runningCount: Option[Int]): Option[Int] = {
    val newCount = runningCount.getOrElse(0) + newValues.sum
    Some(newCount)
  }

    // Set checkpoint directory
    ssc.checkpoint("E:\\sparkdata")

    // Create a DStream that will connect to hostname:port, like localhost:9999
    val lines = ssc.socketTextStream("localhost", 9999)

    // Split each line into words
    val words = lines.flatMap(_.split(" "))

    // Count each word in each batch
    val pairs = words.map(word => (word, 1))

    // Update state using `updateStateByKey`
    val runningCounts = pairs.updateStateByKey[Int](updateFunction _)

    // Print the first ten elements of each RDD generated in this DStream to the console
    runningCounts.print()

    ssc.start() // Start the computation
    ssc.awaitTermination() // Wait for the computation to terminate

And my pom.xml file :///

<dependencies>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka_2.11</artifactId>
        <version>1.6.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>1.6.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>1.6.0</version>
    </dependency>

    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-csv_2.11</artifactId>
        <version>1.2.0</version>
    </dependency>




    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.1.1</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-twitter_2.11</artifactId>
        <version>1.6.0</version>
    </dependency>

    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-stream</artifactId>
        <version>3.0.3</version>
    </dependency>

    <!-- JDBC Connector Jar    -->

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.31</version>
    </dependency>
    <dependency>
        <groupId>au.com.bytecode</groupId>
        <artifactId>opencsv</artifactId>
        <version>2.4</version>
    </dependency>

</dependencies>
</project>
1
  • Would be easier to trace if you provide the full stacktrace. Commented Apr 21, 2018 at 7:17

1 Answer 1

1

you are including different versions of spark.
you have both 2.1.1 and 1.6.0 .

you should use the same spark version for all spark dependencies

Sign up to request clarification or add additional context in comments.

1 Comment

Glad to help. If this answer solved your problem, please mark it as answered, to help future reader know that the question been solved. see here: stackoverflow.com/help/someone-answers

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.