How to use custom user-defined functions across Jupyter Notebooks in EMR clusters?

Question

I want to share udfs I created in Scala with other cluster which our data scientist use with pyspark and jupyter in EMR clusters.

Is this possible? How?

@zeapo Don't think so as it's about sharing UDFs in Jupyter across EMR cluster that could give a feature like this. It's not possible in Spark directly unless people use shared SparkSession in Spark Thrift Server though. — Jacek Laskowski
– Jacek Laskowski, Commented Jul 3, 2017 at 9:46
It's not, because I want to be able to share existing function and add them to the spark catalog, instead of recreate them every time — Lior Baber
– Lior Baber, Commented Jul 3, 2017 at 9:47
Do you want to share the same UDF across different EMR clusters (which I believe are therefore different SparkContexts)? Unless EMR somehow gives you the UDF sharing feature it's not possible in Spark SQL. — Jacek Laskowski
– Jacek Laskowski, Commented Jul 3, 2017 at 9:50
Isn't there something similar to a shared hive metastore? Or add something to spark-default.conf file? — Lior Baber
– Lior Baber, Commented Jul 3, 2017 at 9:51

Lior Baber · Accepted Answer · 2017-07-03 10:06:46Z

1

this answer indeed helps

create an uber jar, put in s3, on bootstrap action copt it from s3 to spark local jar folder and it should work

answered Jul 3, 2017 at 10:06

Lior Baber

8823 gold badges12 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1