Difference between spark-submit vs. SparkSession in python script?

Question

Are there pros/cons, or maybe different use cases for using spark-submit to submit a python script vs. simply running a .py file with the python executable (and importing SparkSession), like this?

from pyspark.sql import SparkSession
spk = SparkSession.builder.master(master).getOrCreate()

Basically, are there any differences running the script via python and not spark-submit.

Possible duplicate of What is the difference between spark-submit and pyspark? — vmg
– vmg, Commented Jun 1, 2017 at 15:59
pyspark runs inside a spark shell, yeah? in this case, i just want to run the script via python and not spark-submit. — Luke W
– Luke W, Commented Jun 1, 2017 at 16:03

Luke W · Accepted Answer · 2017-07-26 20:23:10Z

9

spark-submit is mostly a convenience method. It allows you to set all desired configuration, environment variables, and other options on submit.

It also allows you to set JVM options, which cannot be set on the running virtual machine. Since JVM is initialized once Spark configuration is created, it is not possible to do the same from the running Python process.

edited Jul 26, 2017 at 20:23

Luke W

9,1445 gold badges47 silver badges38 bronze badges

answered Jun 1, 2017 at 16:15

user8098908

1711 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Luke W Over a year ago

after running side-by-side, it also appears that with spark-submit, logging is more verbose by default, and spark-submit also handles cleanup chores, both on failure and success.

Collectives™ on Stack Overflow

Difference between spark-submit vs. SparkSession in python script?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related