Modifying yield from's return value

Question

Let's say I have these parsers:

parsers = {
    ".foo": parse_foo,
    ".bar", parse_bar
}

parse_foo and parse_bar are both generators that yield rows one by one. If I wish to create a single dispatch function, I would do this:

def parse(ext):
    yield from parsers[ext]()

The yield from syntax allows me to tunnel information easily up and down the generators.

Is there any way to maintain the tunneling while modifying the yield results?
Doing so while breaking the tunneling is easy:

def parse(ext):
    for result in parsers[ext]():
        # Add the extension to the result
        result.ext = ext
        yield result

But this way I can't use .send() or .throw() all the way to the parser.

The only way I'm thinking of is by doing something ugly like try: ... except Exception: ... and pass the exceptions up, while doing the same for .send(). It's ugly, messy and bug-prone.

I think your best bet would probably be to implement a passthrough_map that does what map does while passing send and throw through to the generator you're mapping over. IIRC, doing that right is tricky, but you only need to get it right once, and then you can reuse it whenever you need that functionality. — user2357112
– user2357112, Commented Apr 12, 2016 at 20:37

Tamas Hegedus · Accepted Answer · 2016-04-12 21:18:52Z

2

There is another way doing this besides try ... yield ... except: by implementing a new generator. With this class you can transform all the inputs and outputs of your underlying generator:

identity = lambda x: x
class map_generator:
  def __init__(self, generator, outfn = identity,
      infn = identity, throwfn = identity):
    self.generator = generator
    self.outfn = outfn
    self.infn = infn
    self.throwfn = throwfn
    self.first = True
  def __iter__(self):
    return self
  def __next__(self):
    return self.send(None)
  def _transform(self, value):
    if self.first:
      self.first = False
      return value
    else:
      return self.infn(value)
  def send(self, value):
    return self.outfn(self.generator.send(self._transform(value)))
  def throw(self, value):
    return self.outfn(self.generator.throw(self.throwfn(value)))
  def next(self): # for python2 support
    return self.__next__()

Usage:

def foo():
  for i in "123":
    print("sent to foo: ", (yield i))

def bar():
  dupe = lambda x:2*x
  tripe = lambda x:3*x
  yield from map_generator(foo(), dupe, tripe)

i = bar()
print("received from bar: ", i.send(None))
print("received from bar: ", i.send("B"))
print("received from bar: ", i.send("C"))

...

received from bar:  11
sent to foo:  BBB
received from bar:  22
sent to foo:  CCC
received from bar:  33

EDIT: You might want to inherit from collections.Iterator, but it is not neccessary in this usecase.

edited Apr 12, 2016 at 21:18

answered Apr 12, 2016 at 21:11

Tamas Hegedus

30.1k12 gold badges67 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bharel Over a year ago

Thanks very much for the time you took answering me. I found a package called cotoolz that apparently does exactly that, but it is implemented in C thus resulting in faster execution, so I'll go with it.

Zach Gates · Accepted Answer · 2016-04-12 20:28:02Z

0

Have parse_foo and parse_bar add the extensions:

def parse_foo(ext):
    # Existing code
    ...
    # Add an extension to the item(s)
    item.ext = ext

def parse(ext):
    yield from parsers[ext](ext)

Or just hardcode it in each function:

def parse_foo():
    # Existing code
    ...
    # Add an extension to the item(s)
    item.ext = ".foo"

edited Apr 12, 2016 at 20:28

answered Apr 12, 2016 at 20:04

Zach Gates

4,1651 gold badge29 silver badges52 bronze badges

5 Comments

user2357112 Over a year ago

This breaks send and throw.

Bharel Over a year ago

This does not work and indeed breaks send and throw.

Zach Gates Over a year ago

@user2357112 In what way does this break send and throw?

user2357112 Over a year ago

@ZachGates: With yield from, a send or throw on the outer generator gets passed through to the generator you're yield from-ing. Putting a map in the way stops that from working.

Zach Gates Over a year ago

@user2357112: Thanks for the explanation; I believe I've fixed it.

Bharel · Accepted Answer · 2016-04-13 18:23:49Z

0

Unfortunately there is no built-in that does it. You may implement it yourself using classes but a package called cotoolz implements a map() function that does exactly that.

Their map function is 4 times slower than the builtin map() but it's aware to the generator protocol, and faster than a similar Python implementation (it's written in C and requires a C99 compiler).

An example from their page:

>>> def my_coroutine():
...     yield (yield (yield 1))
>>> from cotoolz import comap
>>> cm = comap(lambda a: a + 1, my_coroutine())
>>> next(cm)
2
>>> cm.send(2)
3
>>> cm.send(3)
4
>>> cm.send(4)
Traceback (most recent call last):
    ...
StopIteration

edited Apr 13, 2016 at 18:23

answered Apr 13, 2016 at 18:09

Bharel

27.5k8 gold badges52 silver badges100 bronze badges

Collectives™ on Stack Overflow

Modifying yield from's return value

3 Answers 3

1 Comment

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related