3

I have this code structure in python3:

- datalake
  __init__.py
  utils
     __init__.py
     utils.py
  lambdas
     __init__.py
     my-lambdas.py
- tests
    __init__.py
    demo.py

All init__.py files are empty.

My problem is how I can import datalake module from tests/demo.py? I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.

If I use this code:

from ..datalake.utils import utils

I will get error ValueError: attempted relative import beyond top-level package.

I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.

How can I import the module?

2
  • Is the parent directory (the one containing both datalake and tests) on your PYTHONPATH? I think if it is, it should find the module. Commented Jul 26, 2019 at 1:47
  • 1
    It is not. Do I have to add the path on PYTHONPATH as environment variable? Commented Jul 26, 2019 at 2:09

5 Answers 5

5

When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:

Python 2:

python -m tests/demo

Python 3:

python -m tests.demo

and any datalake imports in demo.py will work.

It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.

Sign up to request clarification or add additional context in comments.

4 Comments

I got this error No module named tests/demo' when run `python -m tests/demo. Do I need to change anything else?
@ZhaoYi are you in the top project folder? so that tests/demo.py exists from the folder you are currently in?
Yes the file tests/demo.py exists in the current directory which is the root of project
@ZhaoYi sorry it seems this syntax only works in python 2. use python -m tests.demo instead.
2

First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8's naming conventions, such as mylambdas.py.

Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils
│   │   ├── __init__.py
│   │   └── utils.py
│   └── lambdas
│       ├── __init__.py
│       └── mylambdas.py
└── tests
    ├── __init__.py
    └── demo.py

Then this should work:

~/myproject$ python -c 'from datalake import utils'

Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.

Also some general advice: I've found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils.py
│   └── lambdas.py
└── tests
    ├── __init__.py
    └── demo.py

3 Comments

How do I run the tests/demo.py from command line? If I am currently in myproject directory, run python -m tests/demo gives me No module named tests/demo error.
If these are unit tests, the unit testing framework often has a runner that discovers these. For example with pytest (which is great, btw), you just run pytest from myproject/ and it will discover appropriately named files (but they should be named something like test_lambdas.py). If demo.py is something you want to interactively run, try python -m tests.demo. Note the dot (.) instead of slash (/) in the module path.
They are not unit tests. they are demo code to run our project from command line.
1

You can add the module directory into your sys.path:

import sys
sys.path.append("your/own/modules/folder")  # like sys.path.append("../tests")

but this is a one-shot method, which is just valid at this time, the added path is not permanent, it will be eliminated after the code completed execution.

Comments

0

One of the ways to import the file directly instead of using from, like import util

Comments

-1

you can try run :

python -m datalake.lambda.my-lambda

follow: https://docs.python.org/3.7/using/cmdline.html#cmdoption-m

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.