Importing python modules from different directories

Question

I have a flask app with essentially the following structure:

app/
  __init__.py
  myapp.py
  common/
    tool1.py
    tool2.py
  web/
    __init__.py
    views.py
    api/
      api_impl.py
  worker/
    __init__.py
    worker.py
    tasks.py

I initialize in myapp.py an important object I use in several places and I can access it from common/tool1.py and web/api/api_impl.py with from myapp import object. I've been able to use tool1 and tool2 in multiple places in web/ and myapp.py importing with from common.tool1 import tool1_def.

Other relevant facts are in myapp.py there is an import web statement for the blueprints and app/__init__.py and worker/__init__.py are empty. web/__init__.py contains the blueprint definitions for the routes.

I can run the app with gunicorn with no issues, but when I try to run my worker with python app/worker/worker.py I get the error ModuleNotFoundError: No module named 'myapp'. The worker.py is trying to import the same object defined in myapp.py.

I just don't understand why I can run the app and it works but when I try to run the worker it doesn't! I'm definitely not fully understanding the import system in this case and everything I've read online doesn't seem to fully clarify this.

Lior Cohen · Accepted Answer · 2019-10-31 00:25:32Z

1

your working imports imply that the project root is the app folder. As such you need to lunch your worker from this folder (or add it to PYTHONPATH environment variable)

python worker/worker.py

Or

python -m worker.worker

In addition your __init__.py in the app folder should be removed as app is not a package but a project root.

answered Oct 31, 2019 at 0:25

Lior Cohen

5,7202 gold badges18 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

gabrielh Over a year ago

Thanks. I removed init.py and it didn't change anything, which I suspect is good. setup.py is in the parent folder from app but it does say the packages are app.web and app.worker (now that I removed init.py). The worker ran successfully with python -m worker.worker but not with python worker/worker.py from the app folder. Any idea why?

gabrielh Over a year ago

One thing, after removing __init__.py from the app folder when I run setup.py it doesn't find any of my packages anymore. Should I move setup.py inside the app folder or manually define them?

Lior Cohen Over a year ago

you need to decide what is your project structure. If you want app to be a package it is fine but then all the imports should be like: import app.worker.worker

Lior Cohen Over a year ago

there are other more advanced ways like relative imports but from my experience imports from the root package downwards is the best practice.

s3dev · Accepted Answer · 2019-10-31 08:40:38Z

One method we use regularly for development and debugging (but also works in production) is the following. You can add this to each (offending?) module, or all modules if you choose.

import os
import sys

# Add your project root to sys.path.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))))

from common import tool1
from common import tool2
from worker import worker
etc ...

Here you are adding your project root to sys.path. When performing imports, Python starts by looking through sys.path for the module you're trying to import. So here, you are adding your project root as the first item in sys.path.

Another advantage is these imports are explicit to your project. For example, if you have a local workers module which you want to import, but also have a workers package in site-packages, this will look in your local project first.

Collectives™ on Stack Overflow

Importing python modules from different directories

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related