PySpark spark-submit command with --files argument Error

Question

I am running a PySpark job in Spark 2.3 cluster with the following command.

spark-submit 
--deploy-mode cluster 
--master yarn 
--files ETLConfig.json 
PySpark_ETL_Job_v0.2.py

ETLConfig.json has a parameter passed to the PySpark script. I am referring this config json file in the main block as below:

configFilePath = os.path.join(SparkFiles.getRootDirectory(), 'ETLConfig.json')
with open(configFilePath, 'r') as configFile:
    configDict = json.load(configFile)

But, the command throws the following error.

No such file or directory: u'/tmp/spark-7dbe9acd-8b02-403a-987d-3accfc881a98/userFiles-4df4-5460-bd9c-4946-b289-6433-drgs/ETLConfig.json'

May I know what's wrong with my script? I also tried with SparkFiles.get() command but it also didn't work.

you need to use sparkContext.addPyFile(filename). It would probably also help if the file was on s3. — Tim
– Tim, Commented Oct 22, 2018 at 21:10
addPyFile is just addFile that also adds it to the PYTHONPATH. Also, in the OP's case, he is trying to read the file from the driver, so adding it from the driver doesn't buy him anything. The driver would already have access at that point! — Ryan Widmaier
– Ryan Widmaier, Commented Oct 25, 2018 at 19:45

user10533328 · Accepted Answer · 2018-10-20 14:42:21Z

5

+25

You use cluster deploy mode. In this case --files path refers not to local path on the machine you use to submit, but to the local path on the worker that is used to spawn the driver, which is an arbitrary node in your cluster.

If you want to distribute files with cluster mode you should store these in a storage that can accessed by each node. You can for example use:

HTTP / HTPPS URL.
HDFS URL.

answered Oct 20, 2018 at 14:42

user10533328

922 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ryan Widmaier Over a year ago

--files specifies a path to a file on the machine submitting the job, not the Yarn node where the driver ends up running.

Ryan Widmaier · Accepted Answer · 2018-10-26 15:02:57Z

3

You should be able to just load it from your PWD in the running driver. Yarn will start the master container process in the same folder as where --files will dump the file. For client mode that would be different, but for cluster mode it should work fine. For example, this works for me:

driver.py

from pyspark import SparkContext, SparkFiles
import os

with SparkContext() as sc:
    print "PWD: " + os.getcwd()
    print "SparkFiles: " + SparkFiles.getRootDirectory()
    data = open('data.json')
    print "Success!"

spark submit

spark-submit --deploy-mode cluster --master yarn --files data.json driver.py

Updated (comparing paths):

I updated my code to print both the PWD (which worked) and SparkFiles.getRootDirectory (which didn't work). For some reason the paths differ. I'm not sure why that is.. but loading files directly from the PWD is what I have always done for accessing files from the driver.

This is what paths printed:

PWD: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/rwidmaier/appcache/application_1539970334177_0004/container_1539970334177_0004_01_000001
SparkFiles: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/rwidmaier/appcache/application_1539970334177_0004/spark-e869ac40-66b4-427e-a928-deef73b34e40/userFiles-a1d8e17f-b8a5-4999-8

Update #2

Apparently, the way it works is --files and it's brethren only guarantee to provide the files in the SparkFiles.get(..) folder on the Executors, not on the Driver. HOWEVER, in order to ship them to the executors, Spark downloads them first to the PWD on the driver, which allows you to access it from there.

It actually only mentions the executors in the help text, not the driver.

  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor. File paths of these files
                              in executors can be accessed via SparkFiles.get(fileName).

For reference, here is where the files are downloaded to the driver.

edited Oct 26, 2018 at 15:02

answered Oct 25, 2018 at 19:31

Ryan Widmaier

8,6232 gold badges33 silver badges33 bronze badges

4 Comments

AngiSen Over a year ago

hi Ryan... where did you keep the file 'data.json' while executing the script using spark-submit in cluster mode?

Ryan Widmaier Over a year ago

It should be a path to a local file on the system you are running spark-submit from. It could be part of your cluster, or just have access to submit to your cluster.

AngiSen Over a year ago

thanks Ryan.. it works.. command used : <spark-submit --deploy-mode cluster --master yarn --files ETLConfig.json PySpark_ETL_Job1.py> I placed both .py and .json in local node and executed spark-submit from that path where these files are kept.... When executed, it copies the files to the name node in HDFS at the time of execution..

Vikrant Singh Rana Over a year ago

@Ryan Widmaier--how to pass --files via python when submitting the jobs to dataproc cluster

Collectives™ on Stack Overflow

PySpark spark-submit command with --files argument Error

2 Answers 2

1 Comment

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related