142

Is there a straightforward way to list the names of all modules in a package, without using __all__?

For example, given this package:

/testpkg
/testpkg/__init__.py
/testpkg/modulea.py
/testpkg/moduleb.py

I'm wondering if there is a standard or built-in way to do something like this:

>>> package_contents("testpkg")
['modulea', 'moduleb']

The manual approach would be to iterate through the module search paths in order to find the package's directory. One could then list all the files in that directory, filter out the uniquely-named py/pyc/pyo files, strip the extensions, and return that list. But this seems like a fair amount of work for something the module import mechanism is already doing internally. Is that functionality exposed anywhere?

13 Answers 13

251

Using python2.3 and above, you could also use the pkgutil module:

>>> import pkgutil
>>> [name for _, name, _ in pkgutil.iter_modules(['testpkg'])]
['modulea', 'moduleb']

EDIT: Note that the parameter for pkgutil.iter_modules is not a list of modules, but a list of paths, so you might want to do something like this:

>>> import os.path, pkgutil
>>> import testpkg
>>> pkgpath = os.path.dirname(testpkg.__file__)
>>> print([name for _, name, _ in pkgutil.iter_modules([pkgpath])])
Sign up to request clarification or add additional context in comments.

12 Comments

This is disturbingly undocumented, but seems like the most correct way to do this. Hope you don't mind I added the note.
pkgutil is there in python2.3 and up actually. Also, while pkgutil.iter_modules() will not work recursively, there is a pkgutil.walk_packages() as well, which will recurse. Thanks for the pointer to this package though.
I can't confirm that pkgutil.walk_packages() recurses, it gives me the same output as pkgutil.iter_modules(), so I think the answer is incomplete.
If anyone cares, this also works perfectly in python 3.8 (Just saying as original post is quiet old)
I don't think you need to use os at all here. You should be able to just do: [name for _, name, _ in pkgutil.iter_modules(testpkg.__path__)]
|
58
import module
help(module)

3 Comments

Although help does list package contents at the bottom of the help text, the question is more along the lines of how to do this: f(package_name) => ["module1_name", "module2_name"]. I suppose I could parse the string returned by help, but that seems more roundabout than listing the directory.
@DNS: help() prints stuff, it doesn't return a string.
I agree this is a roundabout way but it sent me down a rabbit hole to see how help() works. Anyway, the built-in pydoc module can help spit out the string that help() paginates: import pydoc; pydoc.render_doc('mypackage').
24

Maybe this will do what you're looking for?

import imp
import os
MODULE_EXTENSIONS = ('.py', '.pyc', '.pyo')

def package_contents(package_name):
    file, pathname, description = imp.find_module(package_name)
    if file:
        raise ImportError('Not a package: %r', package_name)
    # Use a set because some may be both source and compiled.
    return set([os.path.splitext(module)[0]
        for module in os.listdir(pathname)
        if module.endswith(MODULE_EXTENSIONS)])

10 Comments

I would add 'and module != "init.py"' to the final 'if', since init.py isn't really part of the package. And .pyo is another valid extension. Aside from that, using imp.find_module is a really good idea; I think this is the right answer.
I disagree -- you can import init directly, so why special case it? It sure isn't special enough to break the rules. ;-)
You should probably use imp.get_suffixes() instead of your hand-written list.
Also, note that this doesn't work on subpackages like xml.sax
What is the purpose of [os.path.splitext(module)[0]? First of all, module is not defined... do you mean 'package_name' or 'pathname' (which near as I can tell, are always the same), and then why strip the extension from a package name? Would there ever be one? And for that matter, why include the package at all?
|
17

Don't know if I'm overlooking something, or if the answers are just out-dated but;

As stated by user815423426 this only works for live objects and the listed modules are only modules that were imported before.

Listing modules in a package seems really easy using inspect:

>>> import inspect, testpkg
>>> inspect.getmembers(testpkg, inspect.ismodule)
['modulea', 'moduleb']

6 Comments

I've put imported = import__('myproj.mymod.mysubmod') m = inspect.getmembers(i, inspect.ismodule) but importd path is ~/myproj/__init.py and m is a list with (mymod, '~/myproj/mymod/__init__.py')
@hithwen Don't ask questions in the comments, especially if they're not directly related. Being a good Samaritan: Use imported = import importlib; importlib.import_module('myproj.mymod.mysubmod'). __import__ imports the top-level module, see the documentation.
Hmm, this is promising but it's not working for me. When I do import inspect, mypackage and then inspect.getmembers(my_package, inspect.ismodule) I get an empty list, even though I certainly have various modules in it.
As a matter of fact, this only seems to work if I import my_package.foo and not just import mypackage, in which case it then returns foo. But this defeats the purpose
@user815423426 You are absolutely right ;-) Seems like I was overlooking something.
|
5

This is a recursive version that works with python 3.6 and above:

import importlib.util
from pathlib import Path
import os
MODULE_EXTENSIONS = '.py'

def package_contents(package_name):
    spec = importlib.util.find_spec(package_name)
    if spec is None:
        return set()

    pathname = Path(spec.origin).parent
    ret = set()
    with os.scandir(pathname) as entries:
        for entry in entries:
            if entry.name.startswith('__'):
                continue
            current = '.'.join((package_name, entry.name.partition('.')[0]))
            if entry.is_file():
                if entry.name.endswith(MODULE_EXTENSIONS):
                    ret.add(current)
            elif entry.is_dir():
                ret.add(current)
                ret |= package_contents(current)


    return ret

3 Comments

What's the advantage of using os.scandir as a context manager instead of iterating over the result entries directly?
@monkut See docs.python.org/3/library/os.html#os.scandir which suggest using it as a context manager to ensure that close is called when you are done with it to ensure that any held resources are released.
this doesnt work for re instead it lists every package but adds re. to all of them
3

This should list the modules:

help("modules")

1 Comment

This is not an answer. It prints all available modules, while author asks for modules in a given package.
3

There is a __loader__ variable inside each package instance. So, if you import the package, you can find the "module resources" inside the package:

import testpkg # change this by your package name

for mod in testpkg.__loader__.get_resource_reader().contents():
    print(mod)

You can of course improve the loop to find the "module" name:

import testpkg

from pathlib import Path

for mod in testpkg.__loader__.get_resource_reader().contents():
    # You can filter the name like 
    # Path(l).suffix not in (".py", ".pyc")
    print(Path(mod).stem)
    

Inside the package, you can find your modules by directly using __loader__ of course.

Comments

2

If you would like to view an inforamtion about your package outside of the python code (from a command prompt) you can use pydoc for it.

# get a full list of packages that you have installed on you machine
$ python -m pydoc modules

# get information about a specific package
$ python -m pydoc <your package>

You will have the same result as pydoc but inside of interpreter using help

>>> import <my package>
>>> help(<my package>)

Comments

1

Based on cdleary's example, here's a recursive version listing path for all submodules:

import imp, os

def iter_submodules(package):
    file, pathname, description = imp.find_module(package)
    for dirpath, _, filenames in os.walk(pathname):
        for  filename in filenames:
            if os.path.splitext(filename)[1] == ".py":
                yield os.path.join(dirpath, filename)

Comments

0

The other answers here will run the code in the package as they inspect it. If you don't want that, you can grep the files like this answer

def _get_class_names(file_name: str) -> List[str]:
    """Get the python class name defined in a file without running code
    file_name: the name of the file to search for class definitions in
    return: all the classes defined in that python file, empty list if no matches"""
    defined_class_names = []
    # search the file for class definitions
    with open(file_name, "r") as file:
        for line in file:
            # regular expression for class defined in the file
            # searches for text that starts with "class" and ends with ( or :,
            # whichever comes first
            match = re.search("^class(.+?)(\(|:)", line) # noqa
            if match:
                # add the cleaned match to the list if there is one
                defined_class_name = match.group(1).strip()
                defined_class_names.append(defined_class_name)
    return defined_class_names

Comments

0

To complete @Metal3d answer, yes you can do testpkg.__loader__.get_resource_reader().contents() to list the "module resources" but it will work only if you imported your package in the "normal" way and your loader is _frozen_importlib_external.SourceFileLoader object.

But if you imported your library with zipimport (ex: to load your package in memory), your loader will be a zipimporter object, and its get_resource_reader function is different from importlib; it will require a "fullname" argument.

To make it work in these two loaders, just specify your package name in argument to get_resource_reader :

# An example with CrackMapExec tool

import importlib

import cme.protocols as cme_protocols


class ProtocolLoader:
    def get_protocols(self):
        protocols = {}
        protocols_names = [x for x in cme_protocols.__loader__.get_resource_reader("cme.protocols").contents()]

        for prot_name in protocols_names:
            prot = importlib.import_module(f"cme.protocols.{prot_name}")
            protocols[prot_name] = prot

        return protocols

Comments

0

This one will find all modules in a package without importing the package. It will work for a package installed in `site-packages` or read from source.

Requires Python 3.9+

import importlib.util
import os
import site


def get_modules(package_name):
    try:
        mod_paths = importlib.util.find_spec(package_name).submodule_search_locations
    except AttributeError:
        raise Exception("package not found")
    modules = [package_name]
    for mod_path in mod_paths:
        path = mod_path if "site-packages" in mod_path else package_name
        for dirpath, _, filenames in os.walk(path):
            for filename in filenames:
                if filename.endswith(".py") and not filename.startswith("__"):
                    module_name = (
                        os.path.join(dirpath, filename)
                        .removeprefix(site.getsitepackages()[0])
                        .removeprefix(os.sep)
                        .removesuffix(".py")
                        .replace(os.sep, ".")
                    )
                    modules.append(module_name)
    return sorted(set(modules))

Example usage:

modules = get_modules("requests")
for module in modules:
    print(module)

Comments

-3
def package_contents(package_name):
  package = __import__(package_name)
  return [module_name for module_name in dir(package) if not module_name.startswith("__")]

1 Comment

That only works for modules, not packages. Try it on Python's logging package to see what I mean. Logging contains two modules: handlers and config. Your code will return a list of 66 items, which doesn't include those two names.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.