How to run python spark script with specific jars

Question

I have to run a python script on EMR instance using pyspark to query dynamoDB. I am able to do that by querying dynamodb on pyspark which is executed by including jars with following command.

`pyspark --jars /usr/share/aws/emr/ddb/lib/emr-ddb-hive.jar,/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar`

I ran following python3 script to query data using pyspark python module.

import time
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession, HiveContext

start_time = time.time()
SparkContext.setSystemProperty("hive.metastore.uris", "thrift://nn1:9083")
sparkSession = (SparkSession
                .builder
                .appName('example-pyspark-read-and-write-from-hive')
                .enableHiveSupport()
                .getOrCreate())
df_load = sparkSession.sql("SELECT * FROM example")
df_load.show()
print(time.time() - start_time)

Which caused following runtime exception for missing jars.

java.lang.ClassNotFoundException Class org.apache.hadoop.hive.dynamodb.DynamoDBSerDe not found

How do I convert the pyspark --jars.. to a pythonic equivalent.

As of now I tried copying the jars from the location /usr/share/... to $SPARK_HOME/libs/jars and adding that path to spark-defaults.conf external class path that had no effect.

I think this is what you are looking for.

Luke F
– Luke F

2019-02-08 16:03:31 +00:00
Commented Feb 8, 2019 at 16:03 — Luke F
– Luke F, Commented Feb 8, 2019 at 16:03

Harsh Bafna · Accepted Answer · 2019-02-11 05:13:43Z

3

Use spark-submit command to execute your python script. Example :

spark-submit --jars /usr/share/aws/emr/ddb/lib/emr-ddb-hive.jar,/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar script.py

answered Feb 11, 2019 at 5:13

Harsh Bafna

2,2341 gold badge17 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to run python spark script with specific jars

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related