0

I've been working at this for a good while but I've just been spinning my tires. I have utilities that I want to place into their own repos/packages so they can individually be used, but I also want to be able to install whatever subsection of these utilities with a top level setup.py. Here's my file layout for a top level project I'm calling cool_code:

│
└──── cool_code
    ├── setup.py
    ├── src
    |   ├── __init__.py
    │   └── cool_top_level_stuff.py
    ── pkgA
    │   ├── other_coding_language_folders...
    |   └── python
    |        ├──setup.py
    |        ├──data
    |        ├──docs
    |        ├──examples
    |        ├──test
    |        └──src
    |            ├── __init__.py
    |            └── funkA.py
    ── pkgB
    │   ├── other_coding_language_folders...
    |   └── python
    |        ├──setup.py
    |        ├──data
    |        ├──docs
    |        ├──examples
    |        ├──test
    |        └──src
    |            ├── __init__.py
    |            └── funkB.py
    ──pkgC
    │   ├── other_coding_language_folders...
    |   └── python
    |        ├──setup.py
    |        ├──data
    |        ├──docs
    |        ├──examplesa
    |        ├──test
    |        └──src
    |            ├── __init__.py
    |            └── funkC.py
    └──sub_category
        ├── pkgD
        │   ├── other_coding_language_folders...
        |   └── python
        |        ├──setup.py
        |        ├──data
        |        ├──docs
        |        ├──examples
        |        ├──test
        |        └──src
        |            ├── __init__.py
        |            └── funkD.py
        └── pkgE
            ├── other_coding_language_folders...
            └── python
                 ├──setup.py
                 ├──data
                 ├──docs
                 ├──examples
                 ├──test
                 └──src
                     ├── __init__.py
                     └── funkE.py

I want to be able to import like this with any subset of these packages present, but I don't expect the imports to work if you are missing a sub package:

import cool_code
import cool_code.pkg_A
from cool_code.pkg_B.funkB import funkB
import cool_code.sub_category.pkg_D as pkg_D

This is my current setup.py, but I've been floundering through many iterations. I'm also not sure if I need the py_modules field?:

from setuptools import setup, find_packages, extension,find_namespace_packages
if __name__ == '__main__':
    setup(
        name='cool_code',
        version='1.2.3',
        package_dir={
            # 'cool_code':'src',
            'cool_code.pkg_A': 'pkg_A/python/src/',
            'cool_code.pkg_B': 'pkg_B/python/src/',
            'cool_code.pkg_C': 'pkg_C/python/src/',
            'cool_code.sub_category.pkg_D': 'sub_category/pkg_D/python/src/',
            'cool_code.sub_category.pkg_E': 'sub_category/pkg_E/python/src/',
        },
        packages=[
                # 'cool_code'
                'cool_code.pkg_A',
                'cool_code.pkg_B',
                'cool_code.pkg_C',
                'cool_code.sub_category.pkg_D',
                'cool_code.sub_category.pkg_E',
                ],
        # py_modules=[
        #         'cool_code.pkg_A',
        #         'cool_code.pkg_B',
        #         'cool_code.pkg_C',
        #         'cool_code.sub_category.pkg_D',
        #         'cool_code.sub_category.pkg_E',
        #         ],


        # scripts = [],
    )

There is a lot of stuff I tried that including find_packages to navigate around the src layout. I can't reproduce this, but I got this to kinda work with some permutation of directory layouts and setup.py. The issue was though, I was able to import things like cool_code.pkgA, but in VS code, there was no auto complete when I was following cool_code or cool_code.pkgA. My understanding for this is because python "back propagated" namespaces for cool_code and their subpackages, and VS code's python auto complete only follows through explicit file modules? That is the next problem to solve after figuring out the imports, but it is just as important.

2
  • Did you consider using namespace packages? pkgA, pkgB, pkgC, pkgE, sub_category would be separate installable packages and you would still get the namespaces you want e.g. cool_code.pkg_B.funkB. If you haven't, I can elaborate. Commented Aug 16, 2024 at 9:49
  • I have, and that was the vague "I can't reproduce this" thing I mentioned. I was able to import, but because the top level was a namespace, VS code wasn't able to auto complete (to my knowledge). This is unfortunately a dealbreaker. A big part of this new structure is promoting discoverability of tools with our group. I think the simplest solution for now is to have these packages truly separate, and have their python module name be cool_code_packageA. We will maintain a nice directory structure in our remote repo, but each package will be independent. Thanks for the response though! Commented Aug 16, 2024 at 19:10

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.