Apache Spark providing custom classpath

Question

How do I supply custom classpath to spark app which is running in cluster mode (i.e. driver submits, executors executes)

spark.driver.userClassPathFirst option leads to more classpath conflicts.

--conf spark.executor.extraClassPath=foo/bar.jar --conf spark.driver.extraClassPath=foo/bar.jar

I used this while using spark-submit I don't see this getting in effect.

Does the foo/bar.jar have to be present on executor host or it will also make it available ?

Ravikumar · Accepted Answer · 2018-04-05 02:29:57Z

1

You can use --jars if you want to copy the jars to all executors by job itself. Otherwise you need to copy the jars.

Thanks Ravi

answered Apr 5, 2018 at 2:29

Ravikumar

1,1311 gold badge14 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

--jars foo/bar.jar just this didn't work as expected. should it go with other flags ? thanks

turns out I had to pass both the flags, --jars to make files available and others (mentioned in question) to make them available in classpath.