Can you optimize imports in functions using closures?

Question

These two questions concern using import inside a function vs. at the top of a module. I do not need to be convinced to put my imports at the top, there are good reasons to do that. However, to better understand the technical issues I would like to ask a followup.

Can you get the best of both worlds performance-wise by using a closure and only importing on first run of the function?

Specifically, suppose you have code such as:

import sys
def get_version():
    return sys.version

You want the import to only happen if the function ever gets called, so you move it inside:

def get_version():
    import sys
    return sys.version

But now it is slow if it does get called a lot, so you try something more complex:

def _get_version():
    import sys

    def nested():
        return sys.version

    global get_version
    get_version = nested
    return nested()
get_version = _get_version

Now at least a basic performace test indicates that this last option is slightly slower than the first (taking ~110% as long), but much faster than the second (taking ~20% as long).

First, does this actually work? Do my measurements accurately depict that the second example does more work or is it an artifact of how I measured things.

Second, is there a slowdown from the closure – beyond the first time the function is run?

You can use the lru_cache decorator from functools if it is a peer function. Not sure if it's a great idea or not, but it would cache the result of the function after the first time it's run. — sytech
– sytech, Commented Jan 5, 2017 at 15:34
@sytech: The lru_cache cache test is quite heavy, I doubt it'll give much of a performance boost for something as lightweight as an import (which, past the initial file load, is nothing more than binding a name). — Martijn Pieters
– Martijn Pieters, Commented Jan 5, 2017 at 15:39

Martijn Pieters · Accepted Answer · 2017-01-05 15:42:14Z

3

Closure dereferencing is not any faster than global lookups:

>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=0, releaselevel='final', serial=0)
>>> from timeit import timeit
>>> glob = 'foo'
>>> def f1(): return glob
...
>>> def closure():
...     closed_over = 'bar'
...     def f2():
...         return closed_over
...     return f2
...
>>> f2 = closure()
>>> timeit(f1, number=10**7)
0.8623221110319719
>>> timeit(f2, number=10**7)
0.872071701916866

In addition, even if it were faster, the tradeoff against readability are not worth it, certainly not when faster options are available for when you really need to optimise code.

Locals are the fastest option, always, if you really need to optimise code called from a tight loop, the proper hybrid is using function argument defaults:

import sys.version

def get_version(_sys_version=sys.version):
    return _sys_version

If you are concerned with the impact of the initial file load from an import at startup time, perhaps you should look at the py-demandimport project instead, which postpones loading modules until the first time they are used.

edited Jan 5, 2017 at 15:42

answered Jan 5, 2017 at 15:34

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

otus Over a year ago

Yes, like I wrote in the question my measurements indicated it was slightly slower. The point was to optimize speed while retaining the only-import-if-called property which your function argument example lacks.

otus Over a year ago

This was just hypothetical, but that py-demandimport project seems like something I could actually use. Thanks.

Collectives™ on Stack Overflow

Can you optimize imports in functions using closures?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related