EMR Serverless Airflow Operator not allowing EMR custom images

Question

I want to launch a Spark job on EMR Serverless from Airflow. I want to use Spark 3.3.0 and Scala 2.13 but the 6.9.0 EMR Release ships with Scala 2.12. I created a FAT jar including all Spark dependencies and it won't work either. As an alternative, I am trying to use an EMR custom image by creating an application using --image-configuration with the Airflow operator but it won't just pass through all the arguments from the boto API.

    create_app = EmrServerlessCreateApplicationOperator(
        task_id="create_my_app",
        job_type="SPARK",
        release_label="emr-6.9.0",
        config={"name": "data-ingestion",
                "imageConfiguration": {
                    "imageUri": "xxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/emr-custom-image:0.0.1"}})

Airflow gives the following error message:

Unknown parameter in input: "imageConfiguration", must be one of:
name, releaseLabel, type, clientToken, initialCapacity, maximumCapacity, tags, autoStartConfiguration, autoStopConfiguration, networkConfiguration

This other config won't work either:

config={"name": "data-ingestion",
        "imageUri": "xxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/emr-custom-image:0.0.1"})

Does anybody have any ideas other than downgrading my Scala version?

Hussein Awala · Accepted Answer · 2023-02-10 00:05:55Z

0

Airflow operator passes the argument to the boto3 client, and this client create the application.

The configuration imageConfiguration is added to boto3 client in 1.26.44 (PR), and the other configuration are added in different version (please check the changelog).

So you can try to upgrade the version of boto3 in you Airflow server, provided that it is compatible with the others dependencies, and if not, you may need to upgrade your Airflow version.

answered Feb 10, 2023 at 0:05

Hussein Awala

5,1932 gold badges13 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Oscar Drai Over a year ago

After talking to AWS support I am downgrading Scala to 2.12.15 which is the one compatible with EMR 6.9.0. We'll be anyway creating a custom image to make our jar lighter and so I'll try your suggestion and edit my answer. Thanks for your help.

Collectives™ on Stack Overflow

EMR Serverless Airflow Operator not allowing EMR custom images

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related