Is there anyway to run python script asynchronously in Airflow?

Question

I want to run python script in airflow. To achieve the same I am triggering script using airflow bash operator like below.

sub_dag = BashOperator(
task_id='execute_dependent_dag',
bash_command='python myscript.py',
dag=dag,
trigger_rule="all_success")

However I want it to be triggered asynchronously. Currently it is waiting for script to get finish. I used & as well as nohup to make it run but it didn't work.

Let me know if there is any other way to run it asynchronously. Thanks in advance.

y2k-shubham · Accepted Answer · 2019-02-08 11:47:47Z

1

I believe extending BashOperator to remove wait() call would make that happen with the downside that errors would go silently undetected
Alternatively if the python script / code in question can be imported into your Airflow project, you could try doing the same with PythonOperator through multiprocessing (or a variety of other ways)

Also if you want to get your hands dirty, have a look at this

answered Feb 8, 2019 at 11:47

y2k-shubham

11.9k15 gold badges75 silver badges166 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Is there anyway to run python script asynchronously in Airflow?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related