386

Recently I started playing around with Python and I came around something peculiar in the way closures work. Consider the following code:

adders = [None, None, None, None]

for i in [0, 1, 2, 3]:
   adders[i] = lambda a: i+a

print adders[1](3)

It builds a simple array of functions that take a single input and return that input added by a number. The functions are constructed in for loop where the iterator i runs from 0 to 3. For each of these numbers a lambda function is created which captures i and adds it to the function's input. The last line calls the second lambda function with 3 as a parameter. To my surprise the output was 6.

I expected a 4. My reasoning was: in Python everything is an object and thus every variable is essential a pointer to it. When creating the lambda closures for i, I expected it to store a pointer to the integer object currently pointed to by i. That means that when i assigned a new integer object it shouldn't effect the previously created closures. Sadly, inspecting the adders array within a debugger shows that it does. All lambda functions refer to the last value of i, 3, which results in adders[1](3) returning 6.

Which make me wonder about the following:

  • What do the closures capture exactly?
  • What is the most elegant way to convince the lambda functions to capture the current value of i in a way that will not be affected when i changes its value?

For a more accessible, practical version of the question, specific to the case where a loop (or list comprehension, generator expression etc.) is used, see Creating functions (or lambdas) in a loop (or comprehension). This question is focused on understanding the underlying behaviour of the code in Python.

If you got here trying to fix a problem with making buttons in Tkinter, try tkinter creating buttons in for loop passing command arguments for more specific advice.

See What exactly is contained within a obj.__closure__? for technical details of how Python implements closures. See What is the difference between Early and Late Binding? for related terminology discussion.

Improve this question
17
  • 57
    I have had this problem in UI code. Drove me nuts. The trick is to remember that loops do not create new scope. Commented Jun 24, 2010 at 6:36
  • 4
    @TimMB How does i leave the namespace? Commented May 24, 2013 at 13:46
  • 4
    @detly Well I was going to say that print i wouldn't work after the loop. But I tested it for myself and now I see what you mean - it does work. I had no idea that loop variables lingered after the loop body in python. Commented May 24, 2013 at 19:03
  • 33
    This is in the official Python FAQ, under Why do lambdas defined in a loop with different values all return the same result?, with both an explanation and the usual workaround. Commented Nov 6, 2014 at 1:23
  • 2
    @SteveJessop: See the first paragraph of Lexical environment farther down the page, which explains that in imperative languages, closures have to be "by reference". Python makes this a bit confusing, because it has mutable values which have their own inherent memory location, but it also has assignment statements that don't mutate values but rather mutate the environment—but assuming you want assignment statements to work, the "location" that matters is the name, not the place in memory. Commented Jun 11, 2015 at 21:58

8 Answers 8

Reset to default
337

you may force the capture of a variable using an argument with a default value:

>>> for i in [0,1,2,3]:
...    adders[i]=lambda a,i=i: i+a  # note the dummy parameter with a default value
...
>>> print( adders[1](3) )
4

the idea is to declare a parameter (cleverly named i) and give it a default value of the variable you want to capture (the value of i)

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

+1 for using default values. Being evaluated when the lambda is defined makes them perfect for this use.
+1 also because this is the solution endorsed by the official FAQ.
This is amazing. The default Python behaviour, however, is not.
This just doesn't seem like a good solution though... you are actually changing function signature just to capture a copy of the variable. And also those invoking the function can mess with the i variable, right?
@DavidCallanan we are talking about a lambda: a type of ad-hoc function you typically define in your own code to plug a hole, not something you share through an entire sdk. if you need a stronger signature, you should use a real function.
|
238

What do the closures capture exactly?

Closures in Python use lexical scoping: they remember the name and scope of the closed-over variable where it is created. However, they are still late binding: the name is looked up when the code in the closure is used, not when the closure is created. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

There are at least two ways to get early binding instead:

  1. The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument's default value to the object you want preserved.

  2. More verbosely but also more robustly, we can create a new scope for each created lambda:

    >>> adders = [0,1,2,3]
>>> for i in [0,1,2,3]:
...     adders[i] = (lambda b: lambda a: b + a)(i)
...     
>>> adders[1](3)
4
>>> adders[2](3)
5

    The scope here is created using a new function (another lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:

    def createAdder(x):
    return lambda y: y + x
adders = [createAdder(i) for i in range(4)]
Improve this answer

4 Comments

Python has static scoping, not dynamic scoping.. it's just all variables are references, so when you set a variable to a new object, the variable itself (the reference) has the same location, but it points to something else. the same thing happens in Scheme if you set!. see here for what dynamic scope really is: voidspace.org.uk/python/articles/code_blocks.shtml .
Option 2 resembles what functional languages would call a "Curried function."
Solution 2 is better. I prefer it over the default parameter. It is more logical and less dependent on the specific way Python is designed. A second lambda provides local variables that function like a closure. 👍 👍
Pedantically, closures are the implementation technique which enable lexical scoping to work with late binding in languages where functions are first-class objects, such as Python. Dynamically scoped languages don't have or need them, because they can just do the same dynamic scope resolution they always would - looking up the name by working backwards through the call stack.
56

For completeness another answer to your second question: You could use partial in the functools module.

With importing add from operator as Chris Lutz proposed the example becomes:

from functools import partial
from operator import add   # add(a, b) -- Same as a + b.

adders = [0,1,2,3]
for i in [0,1,2,3]:
    # store callable object with first argument given as (current) i
    adders[i] = partial(add, i) 

print adders[1](3)
Improve this answer

Comments

38

Consider the following code:

x = "foo"

def print_x():
    print x

x = "bar"

print_x() # Outputs "bar"

I think most people won't find this confusing at all. It is the expected behaviour.

So, why do people think it would be different when it is done in a loop? I know I did that mistake myself, but I don't know why. It is the loop? Or perhaps the lambda?

After all, the loop is just a shorter version of:

adders= [0,1,2,3]
i = 0
adders[i] = lambda a: i+a
i = 1
adders[i] = lambda a: i+a
i = 2
adders[i] = lambda a: i+a
i = 3
adders[i] = lambda a: i+a
Improve this answer

4 Comments

It's the loop, because in many other languages a loop can create a new scope.
This answer is good because it explains why the same i variable is being accessed for each lambda function.
I don't think the confusion is because of the loop, because even with a new scope, the value would still change. I think the confusion is because of the lambda - rather, because of the fact that a closure is created for names that the lambda looks up in the enclosing scope. Because Python's functions are first-class objects, it's easy to have the intuition that the lambda should "know" everything it needs to at creation time - like how, when we instantiate a class, __init__ assigns values to self attributes, and self functions as a namespace that is bound early.
Of course, in the example here, we see the flaw in that illustrated by the late lookup of a global. But when I first encountered this problem, years and years ago, I think that I expected the global namespace to be a special case and that local and enclosing namespaces wouldn't behave that way. After all, it's possible to leave those scopes, right? So you'd have to bind early to avoid the problem, right? If adders[1](3) doesn't give 4, why doesn't it raise a NameError for the out-of-scope i if we return the adders and use them elsewhere? The answer, of course, is the closure.
8

Here's a new example that highlights the data structure and contents of a closure, to help clarify when the enclosing context is "saved."

def make_funcs():
    i = 42
    my_str = "hi"

    f_one = lambda: i

    i += 1
    f_two = lambda: i+1

    f_three = lambda: my_str
    return f_one, f_two, f_three

f_1, f_2, f_3 = make_funcs()

What is in a closure?

>>> print f_1.func_closure, f_1.func_closure[0].cell_contents
(<cell at 0x106a99a28: int object at 0x7fbb20c11170>,) 43

Notably, my_str is not in f1's closure.

What's in f2's closure?

>>> print f_2.func_closure, f_2.func_closure[0].cell_contents
(<cell at 0x106a99a28: int object at 0x7fbb20c11170>,) 43

Notice (from the memory addresses) that both closures contain the same objects. So, you can start to think of the lambda function as having a reference to the scope. However, my_str is not in the closure for f_1 or f_2, and i is not in the closure for f_3 (not shown), which suggests the closure objects themselves are distinct objects.

Are the closure objects themselves the same object?

>>> print f_1.func_closure is f_2.func_closure
False
Improve this answer

1 Comment

NB The output int object at [address X]> made me think the closure is storing [address X] AKA a reference. However, [address X] will change if the variable is reassigned after the lambda statement.
1

In answer to your second question, the most elegant way to do this would be to use a function that takes two parameters instead of an array:

add = lambda a, b: a + b
add(1, 3)

However, using lambda here is a bit silly. Python gives us the operator module, which provides a functional interface to the basic operators. The lambda above has unnecessary overhead just to call the addition operator:

from operator import add
add(1, 3)

I understand that you're playing around, trying to explore the language, but I can't imagine a situation I would use an array of functions where Python's scoping weirdness would get in the way.

If you wanted, you could write a small class that uses your array-indexing syntax:

class Adders(object):
    def __getitem__(self, item):
        return lambda a: a + item

adders = Adders()
adders[1](3)
Improve this answer

1 Comment

Chris, of course the above code have nothing to do with my original problem. It's constructed to illustrate my point in a simple way. It is of course pointless and silly.
0

create adder in a function to capture the value:

def create_adder(i):
    return lambda a: i + a


if __name__ == '__main__':
    adders = [None, None, None, None]

    for i in [0, 1, 2, 3]:
        adders[i] = create_adder(i)

    print(adders[1](3))
Improve this answer

Comments

-1

One way to sort out the scope of i is to generate the lambda in another scope (a closure function), handing over the necessary parameters for it to make the lambda:

def get_funky(i):
    return lambda a: i+a

adders=[None, None, None, None]

for i in [0,1,2,3]:
   adders[i]=get_funky(i)

print(*(ar(5) for ar in adders))

giving 5 6 7 8 of course.

Improve this answer

2 Comments

There were already multiple answers showing this technique. I don't understand what this answer is supposed to add.
There were not multiple answers using this. On careful review of the other answers I see it mentioned at the end of Mark Shawabkeh's answer.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.