Kafka with Spark 2.1 Structured Streaming - cannot deserialize

Question

With Apache Spark version 2.1, I would like to use Kafka (0.10.0.2.5) as source for Structured Streaming with pyspark.

In the Kafka topic, I have json messages (pushed with Streamsets Data Collector). However, I am not able to read it with following code:

kafka=spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers","localhost:6667") \
.option("subscribe","mytopic").load()
msg=kafka.selectExpr("CAST(value AS STRING)")
disp=msg.writeStream.outputMode("append").format("console").start()

It generates this error :

 java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer

I tried to add at the readStream line:

.option("value.serializer","org.common.serialization.StringSerializer")
.option("key.serializer","org.common.serialization.StringSerializer")

But it does not solve the problem.

Any idea ? Thank you in advance.

try this on org.apache.kafka.common.serialization.StringDeserializer for both key and value deserializer — Akash Sethi
– Akash Sethi, Commented Mar 30, 2017 at 6:48
take a help from here hope this is what you looking for https://github.com/akashsethi24/Spark-Kafka-Stream-Example/blob/master/src/main/scala/KafkaConsumer.scala — Akash Sethi
– Akash Sethi, Commented Mar 30, 2017 at 6:49
It tried but it does not help. Maybe because I am using Structured Streaming. — JS G.
– JS G., Commented Mar 31, 2017 at 12:19

JS G. · Accepted Answer · 2017-03-31 12:43:08Z

6

Actually I found the solution: I added the following jar in dependency:

spark-streaming-kafka-0-10-assembly_2.10-2.1.0.jar

(after having downloaded it from https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10-assembly_2.10/2.1.0)

answered Mar 31, 2017 at 12:43

JS G.

1581 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ozw1z5rd Over a year ago

It works perfectly! pyspark --jars spark-sql-kafka-0-10_2.11-2.3.2.jar,spark-streaming-kafka-0-10-assembly_2.11-2.3.2.jarallows me to start my test.

Collectives™ on Stack Overflow

Kafka with Spark 2.1 Structured Streaming - cannot deserialize

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related