Yielding from within with statement and exit method of context manager

Question

Consider following Python 2.x code snippet.

from __future__ import print_function


class myfile(file):
    def __exit__(self, *excinfo):
        print("__exit__ called")
        super(myfile, self).__exit__(*excinfo)


def my_generator(file_name):
    with myfile(file_name) as fh:
        for line in fh:
            yield line.strip()


gen = my_generator('file.txt')
print(next(gen))
print("Before del")
del gen
print("After del")

Output of this script (given file.txt has more than one line) is:

Line 1 from file
Before del
__exit__ called
After del

I'm interested about __exit__ call specifically.

What triggers execution of his method? For what we know, code never left with statement (it "stopped" after yield statement and never continued). Is it guaranteed that __exit__ will be called when reference count of generator drops to 0?

Second next call was a leftover from not-yet-minimal example. I fixed code snippet, it's accurate now. — Łukasz Rogalski
– Łukasz Rogalski, Commented May 16, 2017 at 16:08
It would seem so: semantically it's similar to having a finally: python.org/dev/peps/pep-0343 — Simeon Visser
– Simeon Visser, Commented May 16, 2017 at 16:12
not reference count. it's not a garbage collection thing. it happens as soon as you are done with the code block, and all functions called from the code block. I use it all the time to close database connections - it's kind of a pain to pass the connection object around in function calls, but it works very well at making sure you always close them. you could try to fh2 = fh to see if reference count enters into it, but it shouldn't. — JL Peyret
– JL Peyret, Commented May 16, 2017 at 16:37
@JLPeyret: In this case, it actually is a garbage collection thing. — user2357112
– user2357112, Commented May 16, 2017 at 17:35

user2357112 · Accepted Answer · 2017-05-16 17:10:47Z

4

On reclamation of a generator object, Python calls its close method, raising a GeneratorExit exception at the point of its last yield if it wasn't already finished executing. As this GeneratorExit propagates, it triggers the __exit__ method of the context manager you used.

This was introduced in Python 2.5, in the same PEP as send and yield expressions. Before then, you couldn't yield inside a try with a finally, and if with statements had existed pre-2.5, you wouldn't have been able to yield inside one either.

answered May 16, 2017 at 17:10

user2357112

286k32 gold badges490 silver badges571 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:18:26Z

1

To add to @user2357112's answer, the with block breaks when an exception is raised inside of it. This exception is passed to the __exit__ method of the object that was created for the context.

The file class seems to pass silently the GeneratorExit exception, since nothing signals it. However, if you print argc in your myfile.__exit__ method, you will see that the context was not closed naturally:

class myfile(file):
    def __exit__(self, *excinfo):
        print("__exit__ called")
        print(excinfo[0]) # Print the reason why the context exited
        super(myfile, self).__exit__(*excinfo)

Output of your script:

Line 1 from file
Before del
__exit__ called
<type 'exceptions.GeneratorExit'>
After del

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered May 16, 2017 at 17:01

Right leg

17k8 gold badges56 silver badges90 bronze badges

Collectives™ on Stack Overflow

Yielding from within with statement and exit method of context manager

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related