Run script from within pyspark shell

Question

I can run my python+pyspark script from the unix command line by typing

pyspark script.py

But how do I run script.py from within the pyspark shell? This seems like an elementary question but I can't find the answer anywhere. I tried

execfile('script.py')

But I get an error which includes:

ValueError: Cannot run multiple SparkContexts at once

Try subprocess.call(docs.python.org/3/library/subprocess.html#subprocess.call). My understanding is that execfile tries to evaluate the file in the same Python instance, whereas with the subprocess module you can spawn another instance of Python and PySpark, without any conflict. — KartikKannapur
– KartikKannapur, Commented Jul 15, 2016 at 4:20
Thank you for the tip. After making my code executable and adding a chmod, I am able to run the code this way. But after it runs, I cannot access the variables in the code. So it is nearly the same as running 'pyspark script.py' in unix. — user3433489
– user3433489, Commented Jul 18, 2016 at 13:52
Yes, you would not have access to the variables unless you pipe them into another variable or persist them in some data structure. subprocess will only help you invoke another spark program. You could try something similar to subprocess.Popen with stdout=PIPE — KartikKannapur
– KartikKannapur, Commented Jul 18, 2016 at 14:02
I'm curious as to which version of Spark you're using such that you could execute $ pyspark script.py. In my case: "Running python applications through 'pyspark' is not supported as of Spark 2.0." — Quetzalcoatl
– Quetzalcoatl, Commented May 9, 2017 at 20:56

Thomas W · Accepted Answer · 2016-08-24 13:46:04Z

5

Could the error come from script.py trying to create a new SparkContext variable?

When you launch the pyspark interactive client it usually says : SparkContext available as sc, HiveContext available as sqlContext.

If your script file contains sc = SparkContext(), maybe try commenting it.

answered Aug 24, 2016 at 13:46

Thomas W

2833 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1