Failed to find data source: Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide"

Question

Hello I am trying to use pyspark + kafka in order to do this I execute this command in order to set up the kafka-cluster

zookeeper-server-start.sh $KAFKA_HOME/../config/zookeeper.properties

kafka-server-start.sh $KAFKA_HOME/../config/*-0.properties & kafka-server-start.sh $KAFKA_HOME/../config/*-1.properties

Spark version is - spark-3.2.0-bin-hadoop2-7
Kafka version is - kafka_2.13-3.0.0
pyspark version is 3.2.0

The python code is:

spark_version = '3.2.0'
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.12:{}'.format(spark_version)

spark = SparkSession \
    .builder \
    .appName("TP3") \
    .getOrCreate()

!spark-submit --class TP3 --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0 TweetCount.ipynb

This return the following error:

Error: Failed to load class TP3.

And when I execute spark.readStream

consumer = KafkaConsumer('topic')
df_kafka = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", 'localhost:9092') \
    .option("subscribe", 'topic') \
    .load()

And I got this error:

Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide".

How can I execute the readstream in order to read from kafka with pyspark?

Thanks

Tlaloc-ES · Accepted Answer · 2022-01-16 00:15:32Z

10

Finally, I solved using the following code at the beginning of the notebook.

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-10_2.12:3.2.0,org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0 pyspark-shell'

answered Jan 16, 2022 at 0:15

Tlaloc-ES

5,34410 gold badges56 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Namballa Mukesh Over a year ago

Could you explain what causes this error? because I've also added spark.jars.packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.0 in the spark-defaults.conf.template file inside /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/conf/ but still getting this error. Anyway your solution cleared the error, but I want to know how?

Tlaloc-ES Over a year ago

I am not sure, but I think that is a dependency error, about the set the version that you need, only that I guess.

Fabio Over a year ago

I was trying out pyspark in PyCharm and I was getting this error. I just added the lines above to my code and it worked. In Java this will be dependencies added to a pom/sbt file. But in python this will go in the submit command I imagine.

Collectives™ on Stack Overflow

Failed to find data source: Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide"

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related