Apache Kafka-Exception : org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;)

Question

When I submit Spark Structured Streaming application, I am getting below error:

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)

Spark version: 2.1.0 Using Scala version 2.11.1, OpenJDK 64-Bit Server VM, 1.8.0_151

Kafka Version: kafka_2.12-0.10.2.1 (Scala version 2.12, Kafka Version 0.10.2.1)

I have included spark-sql-kafka-0-10_2.11-2.1.0.jar & kafka-clients-0.10.2.1.jar files to spark.driver.extraClassPath & spark.executor.extraClassPath

Can anyone help me to find out why I am getting error?

Is it because different Scala version on Spark 2.1.0 (with Scala version 2.11.1) and Kafka 0.10.2.1 (with Scala version 2.12)? if it is then how to resolve it without installing Kafka 0.10.2.1 (with Scala version 2.11)?

Any suggestions?

sam
– sam

2018-03-13 16:23:41 +00:00
Commented Mar 13, 2018 at 16:23 — sam
– sam, Commented Mar 13, 2018 at 16:23

KiranM · Accepted Answer · 2018-05-11 02:02:54Z

1

Include spark-streaming-kafka-0-10_2.11-2.1.0.jar in spark.driver.extraClassPath in spark-submit should solve your problem. Mostly, it is not related to differing scala versions.

Ex:

spark-submit \
      --conf "spark.driver.extraClassPath=/jar/path/spark-streaming-kafka-0-10_2.11-2.2.0.jar" \

answered May 11, 2018 at 2:02

KiranM

1,3241 gold badge11 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Apache Kafka-Exception : org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related