I'm trying to run a spark job using EMR Serverless but the issue is I cannot pass the list of jars and archives to the spark job.
The spark properties section does not seem to allow passing in a comma delimited list.
AWS documentation page clearly says comma separated-list but is not working. https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html
Any suggestion on how to pass a comma separated jars to the EMR Serverless spark properties?
--conf spark.archives=s3://s3-analytics-core-qa/serverless/pii-poc/config/dev-logback.xml#logback.xml,s3://s3-analytics-core-qa/serverless/pii-poc/config/gobi-dev-pii.conf#pii.conf
--conf spark.jars=s3://s3-analytics-core-qa/serverless/pii-poc/libs/log4j-over-slf4j-1.7.32.jar,s3://s3-analytics-core-qa/serverless/pii-poc/libs/logback-classic-1.2.9.jar,s3://s3-analytics-core-qa/serverless/pii-poc/libs/logback-core-1.2.9.jar,s3://s3-analytics-core-qa/serverless/pii-poc/libs/logstash-logback-encoder-7.0.1.jar,s3://s3-analytics-core-qa/serverless/pii-poc/libs/asm-6.1.jar
--packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk-bundle:1.12.210
