How to execute python code and airflow macro in a BashOperator?

Question

I am trying to execute the following task in Airflow:

time_zone = "America/New_York"
t1= BashOperator(
    task_id='example_task',
    bash_command=('script.sh --arg1 hello '
                  f'--arg2 {{ execution_date + timedelta(days=1).astimezone(pytz.timezone({time_zone})) }}'),
    dag=dag)

The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.

My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.

I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like echo {{ execution_date + timedelta(days=1).astimezone(pytz.timezone({time_zone})) }} to see what that macro is actually resolving to. — chris.mclennon
– chris.mclennon, Commented Mar 29, 2019 at 22:01
So you are saying that this {{ execution_date + timedelta(days=1).astimezone(pytz.timezone({time_zone})) }} runs fine in BashOperator, where execution_date is an airflow macro and the rest is python code? — Newskooler
– Newskooler, Commented Apr 1, 2019 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to {{{{ execution_date + timedelta(days=1).astimezone(pytz.timezone('{time_zone}')) }}}}, doing your stringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to change timedelta to macros.timedelta if you get a timedelta is undefined error. — chris.mclennon
– chris.mclennon, Commented Apr 2, 2019 at 23:29

Newskooler · Accepted Answer · 2019-04-01 22:05:47Z

2

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

edited Apr 1, 2019 at 22:05

answered Apr 1, 2019 at 20:47

Newskooler

4,33310 gold badges61 silver badges94 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to execute python code and airflow macro in a BashOperator?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related