How do I load a Python module with custom globals using importlib?

Question

I'm trying to put together a small build system in Python that generates Ninja files for my C++ project. Its behavior should be similar to CMake; that is, a bldfile.py script defines rules and targets and optionally recurses into one or more directories by calling bld.subdir(). Each bldfile.py script has a corresponding bld.File object. When the bldfile.py script is executing, the bld global should be predefined as that file's bld.File instance, but only in that module's scope.

Additionally, I would like to take advantage of Python's bytecode caching somehow, but the .pyc file should be stored in the build output directory instead of in a __pycache__ directory alongside the bldfile.py script.

I know I should use importlib (requiring Python 3.4+ is fine), but I'm not sure how to:

Load and execute a module file with custom globals.
Re-use the bytecode caching infrastructure.

Any help would be greatly appreciated!

Maybe I'm misunderstanding you, but Python's "globals" are actually module-scoped already - each script's bld global will automatically be distinct (unless they explicitly import it from another module, or store it into another module). — jasonharper
– jasonharper, Commented Mar 1, 2017 at 21:27
I figured, but I'm not much of a Python programmer, so I just wanted to make that requirement clear in case it had any effect on the answer. — David Brown
– David Brown, Commented Mar 1, 2017 at 21:32
If you say custom globals, do you want to set them before importing the module? — MB-F
– MB-F, Commented Mar 1, 2017 at 21:42
@kazemakase Yes, I want them to be available when the module's code executes. — David Brown
– David Brown, Commented Mar 1, 2017 at 21:44
@DavidBrown I don't think this is possible, but it will be interesting to see how someone may prove me wrong. There are certainly easier ways to achieve your desired functionality without hacking imports. For example, you could define entry functions in the modules that perform all the work after the import is finished. Regarding __pycache__, I think it is hard-coded although the definition seems to be unused. Can't elaborate an answer on the import stuff now; will check back tomorrow. — MB-F
– MB-F, Commented Mar 1, 2017 at 22:12

MB-F · Accepted Answer · 2017-03-02 09:31:11Z

1

Injecting globals into a module before execution is an interesting idea. However, I think it conflicts with several points of the Zen of Python. In particular, it requires writing code in the module that depends on global values which are not explicitly defined, imported, or otherwise obtained - unless you know the particular procedure required to call the module.

This may be an obvious or slick solution for the specific use case but it is not very intuitive. In general, (Python) code should be explicit. Therefore, I would go for a solution where parameters are explicitly passed to the executing code. Sounds like functions? Right:

bldfile.py

def exec(bld):
    print('Working with bld:', bld)
    # ...

calling the module:

# set bld

# Option 1: static import
import bldfile
bldfile.exec(bld)

# Option 2: dynamic import if bldfile.py is located dynamically
import importlib.util
spec = importlib.util.spec_from_file_location("unique_name", "subdir/subsubdir/bldfile.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
module.exec(bld)

That way no code (apart from the function definition) is executed when importing the module. The exec function needs to be called explicitly and when looking at the code inside exec it is clear where bld comes from.

edited Mar 2, 2017 at 9:31

answered Mar 2, 2017 at 8:15

MB-F

23.8k5 gold badges71 silver badges127 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

David Brown Over a year ago

Thanks! I'll accept this answer, even though it's not the solution I went with, because it's probably what most projects should do (and in fact, this is what WAF does). In my case, I wanted bldfile.py to appear more like a configuration file than a script file in the simplest cases. Compiling a library, for example, should be as simple as a single line: bld.lib('foo', 'foo.cc'). If I ever decide to make this tool usable outside of my C++ project, I would probably refactor it to use your approach.

David Brown · Accepted Answer · 2017-03-02 01:24:38Z

1

I studied importlib's source code and since I don't intend to make a reusable Loader, it seems like a lot of unnecessary complexity. So I just settled on creating a module with types.ModuleType, adding bld to the module's __dict__, compiling and caching the bytecode with compile, and executing the module with exec. At a low level, that's basically all importutil does anyway.

answered Mar 2, 2017 at 1:24

David Brown

36.4k12 gold badges91 silver badges139 bronze badges

1 Comment

MB-F Over a year ago

Interesting solution. I don't know what you did exactly, but you probably don't need to use ModuleType. You can add bld directly to the globals when calling exec on the code object. exec(code, {'bld': bld})

Georgii Dernovyi · Accepted Answer · 2020-07-04 16:03:40Z

0

It is possible to overcome the lack of possibility by using dummy module, which would load its globals .

#service.py 
module = importlib.import_module('userset')
module.user = user 
module = importlib.import_module('config')

#config.py
from userset import *
#now you can use user from service.py

answered Jul 4, 2020 at 16:03

Georgii Dernovyi

491 silver badge8 bronze badges

Collectives™ on Stack Overflow

How do I load a Python module with custom globals using importlib?

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related