I use EMR Notebook connected to EMR cluster. Kernel is Spark and language is Scala. I need some jars that are located in S3 bucket. How can I add jars?
In case of 'spark-shell' it's easy:
spark-shell --jars "s3://some/path/file.jar, s3://some/path/faile2.jar"
Also in scala console I can do
:require s3://some/path/file.jar
Just put that on your first paragraph:
%%configure -f
{
"conf": {
"spark.jars": "s3://YOUR_BUCKET/YOUR_DRIVER.jar"
}
}
spark.jars which points to S3. I'm afraid stack trace doesn't tell me much, since I'm not sure if EMR stack trace matches the open source Spark code lines.
After you start the notebook, you can do this in a cell:
%%configure -f
{
"conf": {"spark.jars.packages": "com.jsuereth:scala-arm_2.11:2.0,ml.combust.bundle:bundle-ml_2.11:0.13.0,com.databricks:dbutils-api_2.11:0.0.3"},
"jars": [
"//path to external downloaded jars"
],
}
If you're trying to automate I'd suggest this:
In your cluster's bootstrap script, copy the jar file from s3 into a readable location, sort of like so:
#!/bin/bash
aws s3 cp s3://path_to_your_file.jar /home/hadoop/
then in your cluster's software settings (in EMR UI on cluster creation) set the classpath properties:
[
{
"Classification": "spark-defaults",
"Properties": {
"spark.driver.extraClassPath": "/home/hadoop/path_to_your_file.jar",
"spark.jars": "/home/hadoop/path_to_your_file.jar"
}
}
]
(you can add extra properties here like spark.executor.extraClassPath or spark.driver.userClassPathFirst) then launch your cluster and it should be available thru imports.
I had to log into the primary node and run spark-shell to see where the import was located (by typing in import com. and pressing tab to auto complete (theres probably an easier way to do this))
then I was able to import and use the class in zeppelin/jupyter
