how to share code between python project and airflow?

Question

We have a python project structure as following, airflow is a new:

├── python
│   ├── airflow
│   │   ├── airflow.cfg
│   │   ├── config
│   │   ├── dags
│   │   ├── logs
│   │   ├── requirements.txt
│   │   └── webserver_config.py
│   ├── shared_utils
│   │   ├── auth
│   │   ├── datadog
│   │   ├── drivers
│   │   ├── entities
│   │   ├── formatter
│   │   ├── helpers
│   │   └── system
...

We have several other package same level as shared_utils, some are common libraries and some are standalone backend services.

We want to keep airflow part independent and meanwhile to benefit the common libraries. We have python folder in PYTHONPATH, python/airflow is in PYTHONPATH as well (currently airflow doesn't import any code from other package).

I am wondering how can I call code from shared_utils in my airflow dags, or how should I organize the project structure to make it possible?

UPDATE:

it seems there is no conflict when I set python and python/airflow both in PYTHONPATH, after add requirements from shared_utils to airflow, it does work as expected.

How to import functions from other projects in Python? stackoverflow.com/questions/14509192/… — Ronin
– Ronin, Commented Jan 3, 2023 at 19:12
How to share code between python projects? stackoverflow.com/questions/48954870/… — Ronin
– Ronin, Commented Jan 3, 2023 at 19:13
@Ronin please see my update, python and python/airflow are both in PYTHONPATH, maybe I should move airflow out of python. — seaguest
– seaguest, Commented Jan 4, 2023 at 8:06

enslaved_programmer · Accepted Answer · 2023-01-03 18:56:17Z

0

I have a project where I have this layout for the project

|
|-- dags/
|---- dag.py
|-- logs/
|-- plugins/
|---- __init__.py
|---- core.py
|-- airflow.cfg

And then I keep the core stuff in the core.py.

When I want to use the code in the core.py file then I will in the dag.py do the following:

from core import <some function>

Note:

This is my airflow.cfg file and it registers the plugins folder so PythonVirtualOperator can find the code in the plugins.

[core]
dags_folder = {AIRFLOW_HOME}/dags


plugins_folder = {AIRFLOW_HOME}/plugins%

TLDR;

So for your case, I would imagine you can do like this in the airflow.cfg:

plugins_folder = {AIRFLOW_HOME}/shared_utils%

answered Jan 3, 2023 at 18:56

enslaved_programmer

2962 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Hussein Awala · Accepted Answer · 2023-01-03 21:47:22Z

0

You can just move your shared_utils to a new folder my_package in python folder, then add the my_package path to your python path:

# in your host
echo export PYTHONPATH="/path/to/python/my_package:$PYTHONPATH" >> ~/.profile
# in airflow docker image
ENV PYTHONPATH="/path/to/python/my_package"

Now you can import from your package in all the python consoles:

from shared_utils.auth import module_x

answered Jan 3, 2023 at 21:47

Hussein Awala

5,1932 gold badges13 silver badges28 bronze badges

Collectives™ on Stack Overflow

how to share code between python project and airflow?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related