Spark 2.1 Structured Streaming - Using Kafka as source with Python (pyspark)

Question

With Apache Spark version 2.1, I would like to use Kafka (0.10.0.2.5) as source for Structured Streaming with pyspark:

kafka_app.py:

from pyspark.sql import SparkSession

spark=SparkSession.builder.appName("TestKakfa").getOrCreate()

kafka=spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers","localhost:6667") \
.option("subscribe","mytopic").load()

I launched the app in the following way:

./bin/spark-submit kafka_app.py --master local[4] --jars spark-streaming-kafka-0-10-assembly_2.10-2.1.0.jar

After having downloaded the .jar from : mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10-assembly_2.10/2.1.0

And I get such error :

[...] java.lang.ClassNotFoundException:Failed to find data source: kakfa. [...]

Similarly, I cannot run the Spark example of integration with Kafka : https://spark.apache.org/docs/2.1.0/structured-streaming-kafka-integration.html

So I wonder where I am wrong or whether Kafka integration with Spark 2.1 using pyspark is actually supported as this page mentioning only Scala and Java as supported language in the version 0.10 makes me doubt : https://spark.apache.org/docs/latest/streaming-kafka-integration.html (But if not supported yet, why an example in Python was published ?)

Darshita · Accepted Answer · 2017-03-21 00:37:40Z

4

You need to use sql-structured streaming jar "spark-sql-kafka-0-10_2.11-2.1.0.jar" instead of spark-streaming-kafka-0-10-assembly_2.10-2.1.0.jar.

answered Mar 21, 2017 at 0:37

Darshita

814 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Michele Piccolini Over a year ago

How do you know what version to specify, and the syntax to use? (e.g. I've seen using : and , to separate the last part: 2.1.0.jar. Or, should I specify the last .jar part or can I leave it out?) And how do you know what repository to specify, if any? (I'm trying to specify these options as --packages in pyspark-sell arguments)

Collectives™ on Stack Overflow

Spark 2.1 Structured Streaming - Using Kafka as source with Python (pyspark)

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related