BashOperator doen't run bash file apache airflow

Question

I just started using apache airflow. I am trying to run test.sh file from airflow, however it is not work.

Following is my code, file name is test.py

import os
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2015, 6, 1),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    # 'queue': 'bash_queue',
    # 'pool': 'backfill',
    # 'priority_weight': 10,
    # 'end_date': datetime(2016, 1, 1),
}

dag = DAG('test', default_args=default_args)

# t1 and t2 are examples of tasks created by instantiating operators
t1 = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag)

create_command = "sh home/ubuntu/test/inst/scripts/test.sh"

if os.path.exists(create_command):
   t2 = BashOperator(
        task_id= 'cllTest',
        bash_command=create_command,
        dag=dag
   )
else:
    raise Exception("Cannot locate {}".format(create_command))

t2.set_upstream(t1)

when I run python ~/airflow/dags/test.py, it doesn't throw any error.

However, When I run airflow list_dag, it throws following error:

[2017-02-15 20:20:02,741] {__init__.py:36} INFO - Using executor SequentialExecutor
[2017-02-15 20:20:03,070] {models.py:154} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags
[2017-02-15 20:20:03,135] {models.py:2040} ERROR - sh home/ubuntu/test/inst/scripts/test.sh
Traceback (most recent call last):
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/airflow/models.py", line 2038, in resolve_template_files
    setattr(self, attr, env.loader.get_source(env, content)[0])
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: sh home/ubuntu/test/inst/scripts/test.sh

I have tried to used How to run bash script file in Airflow for answer, it doesn't work

Where I am making mistake ?

Nitesh Singh · Accepted Answer · 2021-10-29 09:51:34Z

18

Add a space after .sh it should work this is mentioned in the confluence page of airflow

t2 = BashOperator(
task_id='sleep',
bash_command="/home/batcher/test.sh", // This fails with `Jinja template not found` error
#bash_command="/home/batcher/test.sh ", // This works (has a space after)
dag=dag)

edited Oct 29, 2021 at 9:51

answered Mar 3, 2017 at 6:18

Nitesh Singh

8096 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user7126545 · Accepted Answer · 2017-02-16 12:09:28Z

2

use just the script path, without "sh": create_command = "/home/ubuntu/test/inst/scripts/test.sh"

also make sure that "airflow" user has permissions to execute "test.sh" script.

answered Feb 16, 2017 at 12:09

user7126545

2113 silver badges6 bronze badges

Comments

Necravolver · Accepted Answer · 2017-02-15 22:13:33Z

0

Try without the "sh" just set the command to "home/ubuntu/test/inst/scripts/test.sh"

answered Feb 15, 2017 at 22:13

Necravolver

6995 silver badges10 bronze badges

Collectives™ on Stack Overflow

BashOperator doen't run bash file apache airflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related