1

Let's say we have a generator that is indefinite, where new elements can arrive at any moment with significant (up to indefinite) delay.

An example of such generator is tail -F command. In python (omitting various edge cases) it could be implemented as the following:

def tail_follow(file):
    while True:
        line = file.readline()
        if line:
            yield line
        else:
            sleep(1.0)

Obvious problem with this generator is that it may cause caller's thread to sleep forever. Therefore it should provide caller's side a way to break iteration.

The solution I came up is the following:

def tail_follow(file, on_delay_callback):
    while True:
        line = file.readline()
        if line:
            yield line
        else:
            if on_delay_callback():
                break
            else:
                sleep(1.0)

Is this the only way to get this behavior with Python? I know that there is a send function that allows 2-way data transfer, can it be used to make solution more pythonic?

Improve this question

1 Answer 1

Reset to default
4

I don't see the problem with your approach. I think you are almost there. If you change your generator to yield None when there is nothing to return, then you can simply test for that.

See the following example:

#!/usr/bin/python

import sys
import time

tailpath = "./tailfile"
fd = open(tailpath)

def tail_follow(file):
    while True:
        line = file.readline()
        if line:
            yield line
        else:
            yield None


if __name__ == "__main__":
    for line in tail_follow(fd):
        if line:
            print ("found: %s" % (line))
        else:
            # do something useful
            time.sleep(1.0)

Or alternatively:

if __name__ == "__main__":
    mytail = tail_follow(fd)
    while True:
        # do something useful
        time.sleep(1.0)
        # anything to tail ?
        line = next(mytail)
        if line:
            print ("found: %s" % (line))

Jeff Knupp has a nice post about infinite generators.

Improve this answer
6
  • I guess if line: should be if line is None: in __main__. Commented Aug 7, 2015 at 20:11
  • Can you think of any example in standard library (maybe asyncio?) that does it? Commented Aug 7, 2015 at 20:19
  • In both examples the if line: is correct because you want to do something with the result if tail_follow yields something and ignore it when it yields None. You can easily see that it works if you execute touch ./tailfile to create the file to tail and then echo abc>>./tailfile to add characters to it. In a second shell you then call the script and every time you echo something in the first shell to .tailfile, it is printed in the second shell. I'm not familiar with asyncio. Commented Aug 7, 2015 at 20:30
  • I meant if line is not None, sorry. if line will fail if line is empty which isn't desired behavior. Commented Aug 7, 2015 at 21:07
  • Both if line: and if line is not None: behave the same here (1). If the file your tailing contains an empty line, there is still data: line will contain \n and the examples will print an empty line. If you don't want empty lines, then your application can check for them and ignore them. This leaves the generator generic i.e. contains no behaviour or design decisions. (1) There are cases in which the behaviour is different e.g. if the variable would be a number. Commented Aug 7, 2015 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.