2

In airflow is it possible to control the concurrency of the DAG with an UI param?

As far as I know DAG run parameters are only know as task runtime, so it does not look like a good approach.

Controlling the number of concurrent tasks per dag run via UI seems like a simple thing that should be easy to do, but it looks not possible.

Maybe is it possible to create pools and make an UI param for assigning pools?

I have tested to create a param for the user and setting it in the dag and enabling the option render_template_as_native_obj.

@dag(
...
    params={"max_concurrent_shuffles": Param(1, type="integer")}
    render_template_as_native_obj=True,
...
)

And then in the task instantiation using that param to control the max concurrent task instances:

        max_active_tis_per_dagrun="{{ params.max_concurrent_shuffles }}",

This leads to the scheduler crashing, maybe because of string/integer conversion?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.