Using pytest with a src layer

Question

pytest recommends including an additional directory to separate the source code within a project:

my_package
├── src  # <-- no __init__.py on this layer
│   └── my_package
│       ├── __init__.py
│       └── util_module
│           ├── __init__.py
│           └── utils.py
└── tests
    ├── __init__.py
    └── test_util_module
        ├── __init__.py
        └── test_utils.py

Sadly, they say nothing^[1] about how imports in the test code should work in such a case, which work for my IDE just fine in this naive example^[2], but causes the following error with pytest:

~/my_package$ pytest

====================== test session starts ======================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/user/workspace/my_package, inifile:
collected 0 items / 1 errors     
                                                                                                                                                                      
============================ ERRORS =============================
___ ERROR collecting tests/test_util_module/test_utils.py ___
ImportError while importing test module '/home/user/workspace/my_package/tests/test_util_module/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_util_module/test_utils.py:1: in <module>
    from test.test_module.some_file import starify
E   ModuleNotFoundError: No module named 'my_package.util_module'
!!!! Interrupted: 1 errors during collection !!!!!

I can fix the issue by changing the import of the test to

from src.my_package.util_module.utils import starify

but then my IDE complaints about the src part being redundant, so I'd like to keep it out.

^{[1]_{: Not the case any more. As of version 3.7.3, pytest recommends the editable install also featured in @hoefling's answer at the top of its good practices.}}

^{[2]_{: Setup is virtualenv env -p python3.6; source env/bin/activate; pip install pytest}}

If you export PYTHONPATH=".:src/" before running the tests, does that change anything? — Tom Dalton
– Tom Dalton, Commented May 3, 2018 at 13:00
Did you install my_package, i.e. does it contain a setup.py? — Nils Werner
– Nils Werner, Commented May 3, 2018 at 13:37
@NilsWerner No, and no. Should I always do that before testing? I'd rather run unit tests before build in the CI, but I might be wrong. — Arne
– Arne, Commented May 3, 2018 at 13:38
"of course you: 1) need to build the code in order to test it." - In the general case this is not true ,and for a lot of people (myself included) it is not desireable — Tom Dalton
– Tom Dalton, Commented May 3, 2018 at 14:07
@Dogweather are you just venting because the import system bit you, or do you seriously mean that "if python my_code.py works, then python-test-runner should just work too"? Linking library code correctly is a Tricky Problem, and any framework that tries to handle it for you will come with heavy assumptions and restrictions on how your code can be structured which may bite you much harder. — Arne
– Arne, Commented Jan 15, 2024 at 21:18

hoefling · Accepted Answer · 2022-04-13 19:06:00Z

183

Recommended approach for `pytest>=7`: use the `pythonpath` setting

Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:

pyproject.toml example:

[tool.pytest.ini_options]
pythonpath = [
  "src"
]

pytest.ini example:

[pytest]
pythonpath = src

The path entries are calculated relative to the rootdir, thus the src entry adds path/to/project/src directory to sys.path in this case.

Multiple path entries are also allowed: for a layout

repo/
├── src/
|   └── lib.py
├── src2/
|   └── lib2.py
└── tests
    └── test_lib.py

the configuration

[tool.pytest.ini_options]
pythonpath = [
  "src", "src2",
]

or

[pytest]
pythonpath = src src2

will add both lib and lib2 modules to sys.path, so

import lib
import lib2

will both work.

Original answer

Adjusting the PYTHONPATH (as suggested in the comments) is one possibility to solve the import issue. Another is adding an empty conftest.py file in the src directory:

$ touch src/conftest.py

and pytest will add src to sys.path. This is a simple way to trick pytest into adding codebase to sys.path.

However, the src layout is usually selected when you intend to build a distribution, e.g. providing a setup.py with (in this case) explicitly specifying the root package dir:

from setuptools import find_packages, setup


setup(
    ...
    package_dir={'': 'src'},
    packages=find_packages(where='src'),
    ...
)

and installing the package in the development mode (via python setup.py develop or pip install --editable .) while you're still developing it. This way, your package my_package is correctly integrated in the Python's site packages structure and there's no need to fiddle with PYTHONPATH.

edited Apr 13, 2022 at 19:06

answered May 3, 2018 at 13:50

hoefling

67.8k15 gold badges182 silver badges228 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

hoefling Over a year ago

Also, SO has an excellent answer on this topic.

Arne Over a year ago

I tested both solutions and both work =) thanks for the help!

Mayor Mayer Over a year ago

Don't ask me why, but setting pythonpath = src didn't work for me, even though my main.py is directly inside root/src/main.py. What did work is setting pythonpath = .. I would be very glad if anyone could explain this.

NeilG Over a year ago

Using the pythonpath setting in pyproject.toml is best because the setting is kept in the repo and automatically distributed to other developers. If for some reason you think you do need to set PYTHONPATH you can add an export to .venv/bin/activate for seamless support, but note this in your README.rst for other developers.

MaxU - stand with Ukraine Over a year ago

if you have both pytest.ini and pyproject.toml, then pytest.ini files take precedence over other files, even when empty. So having correctly configured [tool.pytest.ini_options] section in the pyproject.toml file should be enough. Then pytest will read configuration from the pyproject.toml file and we don't need thepytest.ini.

|

Alex Reinking · Accepted Answer · 2022-04-02 01:48:31Z

12

As of PyTest 7.0.0, you can now use the pythonpath option to set some default entries in sys.path. This is the most convenient way of going about this.

In pytest.ini:

[pytest]
pythonpath = src/

In pyproject.toml:

[tool.pytest.ini_options]
pythonpath = "src/"

If you use tox, then you should unset this in its settings, so you don't accidentally test the local version, rather than the installed version.

In tox.ini:

[testenv]
commands = pytest ... -o pythonpath=

See the documentation: https://docs.pytest.org/en/7.1.x/reference/reference.html#confval-pythonpath

answered Apr 2, 2022 at 1:48

Alex Reinking

20.7k8 gold badges68 silver badges110 bronze badges

Comments

Patrick Boundy · Accepted Answer · 2020-09-27 13:05:03Z

3

PYTHONPATH updates weren't working for me when using github actions (known prob). Using this pytest-pythonpath install with pytest.ini file worked for me instead:

pip install pytest-pythonpath # accompany with python_path in pytest.ini, so PYTHONPATH is updated with location for modules under test

With this, basic 'pytest' command happily found all tests in subdirs, and found modules under test based on my pytest.ini (set to match source folders in pycharm)

answered Sep 27, 2020 at 13:05

Patrick Boundy

311 bronze badge

1 Comment

David Medinets Over a year ago

You should be able to use python -m pytest ... so that the PYTHONPATH is available to pytest.

snobb · Accepted Answer · 2024-02-08 22:47:55Z

A bit late for the show, but I believe it's just possible to update sys.path in tests/__init__.py and it should work. I believe that is what the plugin does.

eg. in my case the dir structure looks like:

repo/
├── src/
|   └── foo.py
└── tests
    ├── pytest.ini
    ├── conftest.py
    ├── __init__.py  # update sys.path here
    ├── integration
    │   └── ...
    └── unit
        ├── __init__.py
        └── test_foo.py

So in tests/__init__.py I just do this:

import sys

sys.path.append("src")

After this the pytest ./tests just works and IDE seems to be happy as well, with some caviats. Eg. isort in VSCode (Microsoft Python, iSort and Pylance plugins) works correctly sorting the src imports separately as internal, however in vim (ALE, pyright, isort) it mushes them together with external package imports, but I think it's probably down to difference in config that I need to investigate.

Or alternatively setting PYTHONPATH env variable works too:

PYTHONPATH=".:src" pytest ./tests

Collectives™ on Stack Overflow

Using pytest with a src layer

4 Answers 4

Recommended approach for `pytest>=7`: use the `pythonpath` setting

Original answer

7 Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Recommended approach for pytest>=7: use the pythonpath setting

Original answer

7 Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

Recommended approach for `pytest>=7`: use the `pythonpath` setting