How to run bash script file in Airflow

Question

I have a bash script that creates a file (if it does not exist) that I want to run in Airflow, but when I try it fails. How do I do this?

#!/bin/bash
#create_file.sh

file=filename.txt

if [ ! -e "$file" ] ; then
    touch "$file"
fi

if [ ! -w "$file" ] ; then
    echo cannot write to $file
    exit 1
fi

and here's how I'm calling it in Airflow:

create_command = """
 ./scripts/create_file.sh
"""
t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
)

lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 83, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed

Jean-François Fabre · Accepted Answer · 2021-04-04 15:42:46Z

25

From the tutorial this is OK:

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    retries=3,
    dag=dag)

But you're passing a multi-line command to it

create_command = """
 ./scripts/create_file.sh
"""

should be

create_command = "./scripts/create_file.sh "

Moreover, you also have to make sure that you are in the correct directory to avoid cryptic errors. Do it like this for example:

create_command = "./scripts/create_file.sh "
if os.path.exists(create_command):
   t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
   )
else:
    raise Exception("Cannot locate {}".format(create_command))

edited Apr 4, 2021 at 15:42

answered Oct 20, 2016 at 6:34

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

elcomendante Over a year ago

add space after .sh : "./scripts/create_file.sh "

elcomendante Over a year ago

sometimes you might receive error: This fails with Jinja template not found, in order to overcome add space at the end of the script, not sure what drives this behaviour: ref: cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls

Gregory Arenius Over a year ago

@KarolSudol Its because airflow checks the end of the line you pass in and if it ends in .sh it tries to treat it as a template. A space breaks that check and it doesn't treat it as a template. I still can't figure out the why, though.

andrewm4894 Over a year ago

thanks for that space tip - would noooot have figured that out myself

Piyush Ugale Over a year ago

jinga syntax definition says to add spaces in start & end of defined operator like {{ operator }}, rest looks good.

|

Hrvoje · Accepted Answer · 2021-03-25 09:24:34Z

2

From the documentation:

t2 = BashOperator(
    task_id='bash_example',
    # "scripts" folder is under "/usr/local/airflow/dags"
    bash_command="scripts/test.sh",
    dag=dag)

answered Mar 25, 2021 at 9:24

Hrvoje

15.4k11 gold badges104 silver badges121 bronze badges

Collectives™ on Stack Overflow

How to run bash script file in Airflow

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related