Structuring python projects without path hacks

Question

I have a shared python library that I use in multiple projects, so the structure looks like this:

Project1
    main.py <--- (One of the projects that uses the library)
...
sharedlib
    __init__.py
    ps_lib.py
    another.py

Now in each project's main.py I use the following hack to make it work:

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

import sharedlib.ps_lib
...

Is there a way to do it without using this hack? Or is there a better way to organize the projects structure?

Why not just "install" the sharedlib as package? Then you can import it everywhere. — MSeifert
– MSeifert, Commented Aug 16, 2017 at 10:23
@Rahul I don't like this approach because it's not elegant, and in every module in the project you would have "from .. import ..." — Drxxd
– Drxxd, Commented Aug 16, 2017 at 10:23
@MSeifert but then after each small fix in the shared library, I would have to pip install it again — Drxxd
– Drxxd, Commented Aug 16, 2017 at 10:24
Not necessarily, you can install a "symlink" with something like python setup.py develop. That way if you change sharedlib the changes will take effect immediatly (or after an interpreter restart if you're using an interactive interpreter). — MSeifert
– MSeifert, Commented Aug 16, 2017 at 10:26
@MSeifert That's a good idea! Will the symlink work in windows? — Drxxd
– Drxxd, Commented Aug 16, 2017 at 10:33

Community · Accepted Answer · 2020-06-20 09:12:55Z

I think the best way would be to make sharedlib a real package. That means changing the structure a bit:

sharedlib/
    sharedlib/
        __init__.py
        ps_lib.py
        another.py
    setup.py

And using something like this in the setup.py (taken partially from Python-packaging "Minimal Structure"):

from setuptools import setup

setup(name='sharedlib',
      version='0.1',
      description='...',
      license='...',
      packages=['sharedlib'],   # you might need to change this if you have subfolders.
      zip_safe=False)

Then install it with python setup.py develop or pip install -e . when in the root folder of the sharedlib package.

That way (using the develop or -e option) changes to the contents of sharedlib/sharedlib/* files will be visible without re-installing the sharedlib package - although you may need to restart the interpreter if you're working in an interactive interpreter. That's because the interpreter caches already imported packages.

From the setuptools documentation:

Setuptools allows you to deploy your projects for use in a common directory or staging area, but without copying any files. Thus, you can edit each project’s code in its checkout directory, and only need to run build commands when you change a project’s C extensions or similarly compiled files. [...]

To do this, use the setup.py develop command.

(emphasis mine)

The most important thing is that you can import sharedlib everywhere now - no need to insert the sharedlib package in the PATH or PYTHONPATH anymore because Python (or at least the Python where you installed it) now treats sharedlib like any other installed package.

Alan Kavanagh · Accepted Answer · 2017-08-16 10:31:01Z

0

The way we do it is to use bash entry-scripts for the python scripts. Our directory structure would look similar to the following:

/opt/stackoverflow/
                 -> bin
                 -> conf
                 -> lib
                 -> log

Our lib folder then contains all of our sub-projects

/opt/stackoverflow/lib/
                    -> python_algorithms
                    -> python_data_structures
                    -> python_shared_libraries

and then when we want to execute a python script, we'll execute it via a bash script within the bin directory

/opt/stackoverflow/bin/
                    -> quick_sort.sh
                    -> merge_sort.sh

and if we cat one of our entry scripts

cat merge_sort.sh

#!/bin/bash
export STACKOVERFLOW_HOME=/opt/stackoverflow
export STACKOVERFLOW_BIN=${STACKOVERFLOW_HOME}/bin
export STACKOVERFLOW_LIB=${STACKOVERFLOW_HOME}/lib
export STACKOVERFLOW_LOG=${STACKOVERFLOW_HOME}/log
export STACKOVERFLOW_CONF=${STACKOVERFLOW_HOME}/conf

# Do any pre-script server work here

export PYTHONPATH=${PYTHONPATH}:${STACKOVERFLOW_LIB}

/usr/bin/python "${STACKOVERFLOW_LIB}/python_algorithms/merge_sort.py" $* 2>&1

edited Aug 16, 2017 at 10:31

answered Aug 16, 2017 at 10:28

Alan Kavanagh

9,9338 gold badges46 silver badges68 bronze badges

2 Comments

hiro protagonist Over a year ago

doesn't this cause your PYTHONPATH to grow? every time you run this bash script, the path ${STACKOVERFLOW_LIB} is appended (regardless if it is part of the path already).

Alan Kavanagh Over a year ago

Assuming you use the 1 user for everything and dont log out of the server it will continue to grow.

Collectives™ on Stack Overflow

Structuring python projects without path hacks

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related