Airflow-Trigger job using python script

Question

I have list of dags based on categories. These dags should be called from master dag based on priority of categories(to avoid resource constraint).

My thought is to create a python script, where these category dags should be triggered based on priority(not time based) which we set in metadata table. This python script will be called in master dag using bashoperator.

To be clear, Master dag should be time based. Sub dags should be triggered from master based on priority.

My questions are..

1)Is dag can be triggered using python? without time based. 2)I want to run few category dags in parallel. How this can be accomplished. Kindly help me to fix this problem.

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

You can use TriggerDagRunOperator and set priority_weight on each task that uses TriggerDagRunOperator to achieve that.

TriggerDagRunOperator: Triggers a DAG run for a specified dag_id

priority_weight (int) – priority weight of this task against other task. This allows the executor to trigger higher priority tasks before others when things get backed up. Set priority_weight as a higher number for more important tasks.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jul 1, 2019 at 17:02

kaxil

19.1k3 gold badges67 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Airflow-Trigger job using python script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related