What I would like to do is run a docker image in a DockerOperator. While inside the container, I would like to spin up a python interactive shell for a .py file and run a method inside the file.
Is this possible in one operator?
Add a comment
|
1 Answer
I am not sure what you mean by "interactive shell" - but I assume it is only about simply running a new python interpreter (rather than it being interactive and a user providing some code on-the-flight - that would be rather difficult).
Yes. You can put your python file next to your DAG and exclude it from parsing as DAG vi .airflowignore file (using regexps) - you can read more here: https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html
Also just can prepare a custom image with your python file and necessary libraries added.
Here are the instructions on how to do it with plenty of examples: https://airflow.apache.org/docs/docker-stack/build.html
Also - and this is the nicest and newest way - if you are using Airlfow 2.2+, you can use @task.docker operator https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments and in this case you can simply decorate a python callable with @task.docker and the python code will be executed using the docker image you specify (using DockerOperator under the hood)
