spark-submit command with --py-files fails if the driver class path or executor class path is not set

Question

I have a main script as below

from pyspark.sql.session import SparkSession
..............
..............
..............
import callmodule as cm <<<--- This is imported from another pyspark script which is in callmod.zip file
..............
..............
..............

when I submit the spark command as below it fails with Error: No module named Callmodule

spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip mainscript.py

when I submit the spark command with driver class path(without executor class path) as below it runs successfully.

spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip --driver-class-path C:\pyspark\scripts\callmod.zip mainscript.py
                                (or)
spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip --conf spark.driver.extraClassPath=C:\pyspark\scripts\callmod.zip mainscript.py
                                (or)
spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip --conf spark.driver.extraLibraryPath=C:\pyspark\scripts\callmod.zip mainscript.py

when I submit the spark command with executor class path (without driver classpath) also it runs successfully.

spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip --conf spark.executor.extraClassPath=C:\pyspark\scripts\callmod.zip mainscript.py
                               (or)
spark-submit --master local --py-files C:\pyspark\scripts\callmod.zip --conf spark.executor.extraLibraryPath=C:\pyspark\scripts\callmod.zip mainscript.py

Can you explain me where does the below import statement work? on driver or executor?

import callmodule as cm

Why is the code not failing with Error: No Module Named callmodule when only the driver classpath is set or only the executor classpath is set?

mck · Accepted Answer · 2020-12-22 07:37:21Z

1

You are using --master local, so the driver is the same as the executor. Therefore, setting classpath on either driver or executor produces the same behaviour, and neither would cause an error.

answered Dec 22, 2020 at 7:37

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

shankar Over a year ago

even in local mode, will not the driver and executor run on separate jvm? in that case executor wont be able to recognize driver's classpath and viceversa. isn't it?

mck Over a year ago

@shankar I don't think so. In local mode there is only 1 jvm. The driver and executor are the same thing, on the same jvm.

shankar Over a year ago

okay. got it. So If I have to run the spark-submit job with --py-files option on a real cluster, what is the right one use --driver-class-path or executor classpath or both?

mck Over a year ago

@shankar I think you'll need to specify both, so that both driver and executors can find the files.

Collectives™ on Stack Overflow

spark-submit command with --py-files fails if the driver class path or executor class path is not set

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related