Testing with Optional Import in Python

Question

I have a library that can use two modules; one is fast but available only on Linux and macOS, the other is slower but it's multi-platform. My solution was to make the library compatible with both and have something like the following:

try:
    import fastmodule
except ImportError:
    import slowmodule

Now I want to compare the timing of the library when using either module. Is there any way of masking the fastmodule without changing the source code (i.e. within a Jupyter Notebook), in an environment where both modules are installed, so that the slowmodule is used?

MrBean Bremen · Accepted Answer · 2020-03-24 20:34:52Z

1

This is a bit of a hack, but here it goes:
You can write your own importer and register it (note that this is Python 3-specific, Python 2 had another API for this):

import sut
import functools
import importlib
import sys


def use_slow(f):
    @functools.wraps(f)
    def wrapped(*args, **kwargs):
        ImportRaiser.use_slow = True
        if 'fastmodule' in sys.modules:
            del sys.modules['fastmodule']  # otherwise it will remain cached
        importlib.reload(sut)
        f(*args, **kwargs)

    return wrapped


def use_fast(f):
    @functools.wraps(f)
    def wrapped(*args, **kwargs):
        ImportRaiser.use_slow = False
        importlib.reload(sut)
        f(*args, **kwargs)

    return wrapped


class ImportRaiser:
    use_slow = False

    def find_spec(self, fullname, path, target=None):
        if fullname == 'fastmodule':
            if self.use_slow:
                raise ImportError()


sys.meta_path.insert(0, ImportRaiser())

@use_fast
def test_fast():
    # test code


@use_slow
def test_slow():
    # test code

Here, sut is your module under test, which you have to reload in order to change the behavior. I added decorators for readabilty, but this can be done by some function or in the test setup, of course.

If you use the slow version, fastmodule will raise ImportError on import and slowmodule will be used instead. In "fast" case, all works as usual.

edited Mar 24, 2020 at 20:34

answered Mar 23, 2020 at 18:48

MrBean Bremen

17.1k3 gold badges35 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

RMeli Over a year ago

Thanks for the trick, but this does not seem to be called at all...

RMeli Over a year ago

Actually, this works perfectly if I didn't load my package previously. I tried to use importlib.reload but didn't work. Is there a way to make this work within a single script where I want to run with both fastmodule and slowmodule in order to aggregate the data?

MrBean Bremen Over a year ago

I extended the answer a bit to show how to use both slowmodule and fastmodule in your test. Admittedly not the nicest code, but it should work (I tested it with some dummy code).

RMeli Over a year ago

Thanks! del sys.modules['fastmodule'] was the missing step! =)

Collectives™ on Stack Overflow

Testing with Optional Import in Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related