In airflow is it possible to control the concurrency of the DAG with an UI param?
As far as I know DAG run parameters are only know as task runtime, so it does not look like a good approach.
Controlling the number of concurrent tasks per dag run via UI seems like a simple thing that should be easy to do, but it looks not possible.
Maybe is it possible to create pools and make an UI param for assigning pools?
I have tested to create a param for the user and setting it in the dag and enabling the option render_template_as_native_obj.
@dag(
...
params={"max_concurrent_shuffles": Param(1, type="integer")}
render_template_as_native_obj=True,
...
)
And then in the task instantiation using that param to control the max concurrent task instances:
max_active_tis_per_dagrun="{{ params.max_concurrent_shuffles }}",
This leads to the scheduler crashing, maybe because of string/integer conversion?