PySpark Kafka Error: Missing application resource

Question

Below error is triggered when i added the below dependency to the code,

'--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.1.1'

Below is the code,

from pyspark.sql import SparkSession, Row
from pyspark.context import SparkContext
from kafka import KafkaConsumer
import os

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.1.1'


sc = SparkContext.getOrCreate()
spark = SparkSession(sc)

df = spark \
  .read \
  .format("kafka") \
  .option("kafka.bootstrap.servers", "localhost:9092") \
  .option("subscribe", "Jim_Topic") \
  .load()
df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")

Below is the error,

Error: Missing application resource.

Usage: spark-submit [options] [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args]

Giorgos Myrianthous · Accepted Answer · 2020-06-12 16:44:48Z

1

You also need to provide the name of your python file.

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.1.1 your_python_file.py'

Alternatively, a nicer way would be:

conf = SparkConf().set("spark.jars", "/path/to/your/jar")
sc = SparkContext(conf=conf)

answered Jun 12, 2020 at 16:44

Giorgos Myrianthous

40.4k21 gold badges156 silver badges175 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jim Macaulay Over a year ago

I use your alternate wayby providing the path to the jars. Am getting below error

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 20/06/12 22:32:17 ERROR SparkContext: Failed to add /C:/Hadoop/Spark/spark-3.0.0-preview2-bin-hadoop2.7/bin/ to Spark environment java.lang.IllegalArgumentException: Directory C:\Hadoop\Spark\spark-3.0.0-preview2-bin-hadoop2.7\bin is not allowed for addJar

Collectives™ on Stack Overflow

PySpark Kafka Error: Missing application resource

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related