2

Suppose I have an arbitrary function f in Python, that takes parameters.

def f(x): return 2*x

Now suppose I want a function that takes a function and returns the same function, but flipped along the y-axis (if it were graphed).

The obvious way to do it is

def reverse_fn(f): return lambda x, funct=f: funct(-x) 

However, stacking function-modifying functions like this ends up breaking max recursion depth after a while, since the result is just a function that called another function that calls more functions all the way down.

What is the best way to make function-modifying-functions in Python, that can be used over and over again without taking excessive call stack or nesting functions?

3
  • seems like funct will not call another functions that calls more functions all the way down. Commented Apr 3, 2014 at 2:25
  • I used a for loop to apply f = reverse_fn(f) 1000 times. f(1) now promptly throws a RuntimeError: maximum recursion depth exceeded Commented Apr 3, 2014 at 2:30
  • The best way is generally to rework your code to not stack so many decorators. Normal Python code won't apply more than a few to a single function. Like tail recursion, it's a programming style Python doesn't really support. Commented Apr 3, 2014 at 2:51

3 Answers 3

2

One approach is editing the bytecode of the function. This is a very advanced technique, and is also very fragile. So, don't use this for production code!

That said, there is a module out there which implements precisely the kind of editing you want. It's called bytecodehacks, first released on April 1, 2000 (yes, it was an April Fools' joke, but a completely functional one). A slightly later edition (from 2005) works fine on my install of Python 2.7.6; grab it from CVS and run setup.py as usual. (Don't use the April2000 version; it won't work on newer Pythons).

bytecodehacks basically implements a number of utility routines that make it possible to edit the bytecode of a section of code (a function, module, or even just a single block within a function). You can use it to implement macros, for example. For the purposes of modifying a function, the inline tool is probably the most useful.

Here's how you would implement reverse_fn using bytecodehacks:

from bytecodehacks.inline import inline

def reverse_fn(f):
    def g(x):
        # Note that we use a global name here, not `f`.
        return _f(-x)
    return inline(g, _f=f)

That's all! inline takes care of the dirty business of "inlining" the function f into the body of g. In effect, if f(x) was return 2*x, then the return from reverse_fn(f) would be a function equivalent to return 2*(-x) (which would not have any function calls in it).

Now, one limitation of bytecodehacks is that the variable renaming (in extend_and_rename in inline.py) is somewhat stupid. So, if you apply reverse_fn 1000 times in a row, you will get a huge slowdown as the local variable names will begin to explode in size. I'm not sure how to fix this, but if you do, it will substantially improve the performance for functions that are repeatedly inlined.

Sign up to request clarification or add additional context in comments.

Comments

1

The default recursion limit of 1000 can be increased with sys.setrecursionlimit(), but even 1000 is extraordinarily deep recursion, and comes at a steep performance penalty if your wrappers tend to be this kind of trivial alteration you show in your example.

What you could do, if you're trying to build up complex functions procedurally from simple primitives, is to compose the compound functions as Python source text and pass them through eval() to get callable functions. This approach has the significant advantage that a function built up from 1000 primitives won't incur the cost of 1000 function calls and returns when executed.

Note that eval() should be used with caution; don't eval() untrusted sources.

eval() will be fairly expensive per function created, and without knowing a little more about what you're trying to do, it's hard to advise. You could also simply write a program that generates a big .py file full of the compound functions you want.

4 Comments

I have thought of this before, but I think the anti-eval() propaganda from javascript has corrupted me. Obviously this would be better than a 1000-deep recursion, but would eval() have a significant detriment to run speed?
@NameHere: bytecode manipulation is a bit harder to achieve but will result in much better looking code than with eval; or, if you're in a hurry, eval should do just fine :)
eval() is going to be very expensive, but that's a one-time cost per function so created.
with eval you'll have to parse out the names of the variables used in the definition, unless you want to have to pass everything in as kwargs later.
0

I don't think you can achieve this in any language that doesn't support Tail Call Optimization without using a trampoline. Another option is to extract the AST of the function under question and generate a "brand new" function that doesn't call the original function at all but implementing this is not trivial and requires good understanding of some of the more internal parts of Python.

A trampoline on the other hand is easy to implement but has the drawback that your functions cannot be simple Python functions anymore—every time they need to make a recursive call, they return that call as, say, a tuple in the form (some_fn, args, kwargs) (while normal return values would be wrapped in a 1-tuple), the trampoline would then make that call for you so that the stack doesn't grow.

def rec(fn, *args, **kwargs):
    return (fn, args, kwargs)

def value(val):
    return (val,)

def tailrec(fn, *args, **kwargs):
    while True:
        ret = fn(*args, **kwargs)
        if ret is None:
            return None
        elif len(ret) == 1:
            return ret[0]
        else:
            fn, args, kwargs = ret  # no kwargs supported if using tuples

def greet_a_lot(n):
    if n > 0:
        print "hello: " + str(n)
        return rec(greet_a_lot, n - 1)
    else:
        return value("done")

print tailrec(greet_a_lot, 10000)

Output:

hello: 100000
hello: 99999
...
hello: 3
hello: 2
hello: 1
done

1 Comment

The function itself no longer has any AST. You'd have to edit the bytecode. (This is actually likely to be easier, not harder).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.