4

I'm trying to get myself familiar with importlib hooks. I want to implement an ability to directly import non-pythonic files written in other language, and maintain source maps, so raising SyntaxErrors with line numbers will still give meaningful stacktraces.

My approach to loading foreign files is assembling Pythonic source, then compiling it and executing it in the desired context.

I've read in the documentation that implementing importlib.abc.SourceLoader seems to be my choice — however, the only method that is being called is exec_module. At that stage, to my understanding, module object is fully ready. So why are get_source, get_data, get_code and others are not called?

My stub implementation:

import sys
import os

import importlib.abc
import importlib.machinery

class MyFinder(importlib.abc.MetaPathFinder):
    def __init__(self):
        pass

    def find_spec(self, fullname, path, target=None):
        print('find_spec', fullname, path, target)

        # filename = '{}.npy'.format(fullname)
        # if not os.path.exists(filename):
        #     return

        if fullname != 'foobar':
            return

        filename = 'foobar://ponyworld/foo.py'

        spec = importlib.machinery.ModuleSpec(
            name = fullname,
            loader = MyLoader(fullname, path, target),
            origin = filename,
            loader_state = 1234,
            is_package = False,
        )

        return spec

class MyLoader(importlib.abc.SourceLoader):
    def __init__(self, fullname, path, target):
        pass

    def get_data(self, path):
        print('get_data', path)

    def get_filename(self, fullname):
        print('get_filename', fullname)

    def path_stats(self, path):
        print('path_stats', path)

    def set_data(self, path, data):
        print('set_data', path, data)

    def get_code(self, fullname):
        print('get_code', fullname)

    def exec_module(self, module):
        print('exec_module', module)
        print(dir(module))

    def get_source(self, fullname):
        print('get_source', fullname)

    def is_package(self, fullname):
        print('is_package', fullname)

sys.meta_path.append(MyFinder())

# import fake module to see if it works
import foobar
1
  • Since you don't do anything in the Loader's __init__ you can combine MyLoader and MyFinder into MyImporter by decorating find_spec as a classmethod The __init__ then becomes the init for the SourceLoader parent. Supposedly this saves one from instantiating the Loader. Though this makes it a little less flexible if the Finder needs localized information, as this is only really passed in by instantiation. Commented Nov 24, 2016 at 9:39

1 Answer 1

3

I misunderstood documentation and overlooked that importlib.abc.SourceLoader already provides implementations for get_code, exec_module, load_module, get_source and is_package. Those are to be overridden only in need, and only get_data and get_filename are the required minimum.

Sign up to request clarification or add additional context in comments.

1 Comment

Don’t forget about importlib.machinery.SourceFileLoader! It is not an Abstract Base Class, and provides implementations for all of these. (Note the class name: SourceFileLoader. I once tried to replace .abc. with .machinery., because at first glance it’s easy for your brain to assume it has the same class name as the SourceLoader from importlib.abc…but that’s not the case. ☻)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.