40

I have a Python project consisting of a Jupyter notebook, several scripts in a bin directory and modules in a src directory, with dependencies in a Pipfile:

myproject
├── myproject.ipynb
├── Pipfile
├── Pipfile.lock
├── bin
│   ├── bar.py
│   └── foo.py
└── src
    ├── baz.py
    └── qux.py

The scripts foo.py and bar.py use the standard shebang

#!/usr/bin/env python

and can be run with pipenv shell:

mymachine:myproject myname$ pipenv shell
(myproject-U308romt) bash-3.2$ bin/foo.py
foo

However, I can't easily access the modules in src from the scripts. If I add

import src.baz as baz

to foo.py, I get:

ModuleNotFoundError: No module named 'src'

One solution I tried is to add a .env file under myproject:

PYTHONPATH=${PYTHONPATH}:${PWD}

This works thanks to pipenv's automatic loading of .env, but checking the .env file into the git distribution of the project would collide with the traditional use of .env to store secrets such as passwords -- in fact, my default .gitignore for Python projects already excludes .env for just this reason.

$ git add .env
The following paths are ignored by one of your .gitignore files:
.env
Use -f if you really want to add them.

Alternatively, I could move src under bin, but then the Jupyter notebook would have to reference the modules as bin.src.baz etc., which is also a hassle.

My current workaround is just to add a symlink:

myproject
├── Pipfile
├── Pipfile.lock
├── bin
│   ├── bar.py
│   ├── foo.py
│   └── src -> ../src
└── src
    ├── baz.py
    └── qux.py

This works, and I suppose has the benefit of being transparent, but it seems like there should be some way to leverage pipenv to solve the same problem.

Is there a portable, distributable way to put these modules on the search path?

2
  • 2
    I got the same .env idea as you did. But then the value of my path is literally ${PYTHONPATH}:${PWD}. Why did it work for you? Commented Aug 2, 2018 at 11:46
  • @Pranasas I don't know, sorry. Try asking a new question. Commented Aug 2, 2018 at 20:36

4 Answers 4

20

I'm not sure there's a perfect solution for this, but in the interest of being explicit rather than implicit (PEP 20), I've decided to check in a file that needs to be sourced before running any script. This is one extra manual step but you can put this in a Makefile for instance.

env.sh

export PYTHONPATH=${PYTHONPATH}:${PWD}

Makefile

bar:
    source env.sh && pipenv run python scripts/bar.py
.PHONY: migrate

The solution is a bit similar to the approach Go takes with its GOPATH.

I think the other solutions are not as good:

  • pipenv aims to solve dependencies, I could be wrong but I did not find anything related to the problem of the PYTHONPATH.
  • Linking folders won't scale really well if you start having other scripts folder.
Sign up to request clarification or add additional context in comments.

3 Comments

pipenv automatically loads .env files too
Instead of creating a bash script, you can just set the env vars before the command to run pipenv. Using the same line you used in your Makefile: PYTHONPATH=${PYTHONPATH}:${PWD} pipenv run python scripts/bar.py
Not to nitpick but how is this solution (or any similar solution for that matter) easier and more explicit than just running export PYTHONPATH=.? You are making a separate script and creating a Makefile to save one line of code? It looks extremely non-pythonic :)
14

(Came here for an answer, ended up giving one instead)

I have a similar project folder structure, so I had the same problem.
Thanks to your tip, my solution was to add an file .env at the same level as the Pipfile with the following content:

$ cat .env
PYTHONPATH=${PYTHONPATH}:src

Now, launching my app with something like

$ pipenv run python -m package.subpackage.app

seems to work ok from inside my project's folder and also from it's subfolders.

Side note(although is not a good/clean way to do things):
for your ModuleNotFoundError: No module named 'src' problem ... the "problem" is that the src (folder) is not a package, in order to fix that you could easily add an (empty) __init__.py file inside the src folder, thous making it a "package"; which in turn would make import src.baz possible.

(Later edit)
Actually this adds a record <project_folder>/${PYTHONPATH} in sys.path, which is useless, so the correct content of the .env file should be only PYTHONPATH=src.

3 Comments

This is the most convenient non-workaround answer. The .env file (env vars) is in my opinion the best solution.
3 hours later I found your answer, thank you bro, so simple :)
I found this depends on running pipenv run python from the correct folder, since relative paths are used.
0

You probably can use two separate .env-files. The one for your passwords and the one for pipenv somewhere else. You can use a custom path for your pipenv .env-file by setting the PIPENV_DOTENV_LOCATION variable in the environment. It is debateable if you consider this portable, if you have to set the variable first.

Anyway, from experience you should not rely too strong on the autoload of .env. If you are using remote/ssh interpeters or having several environments in parallel for testing purposes the cleanest way would be to just add your project path to a *.pth file in the site-packages folder of your Python environment. In conda there is an explicit "develop" comand for this, in pipenv you still have to do it manually though. Still, this is the way, because you create your Python environment explicitly for your project in an arbitrary setup, and then you also should hardwire the project location into this setup.

Comments

0

In the project root directory, I:

  • created a minimal setup.py.
  • Ran pipenv install -e .

Then anywhere in the project directory I can run pipenv run python file/path.py and imports like import src.baz as baz seem to work.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.