TensorFlow Dataset API from_generator End of Sequence Error

Question

Here is toy code that duplicates a problem I am encountering when trying to generate/feed training data on the fly using a generator.

def makeRand():
    yield np.random.rand(1)

dataset = tf.data.Dataset.from_generator(makeRand, (tf.float32))

iterator = tf.contrib.data.Iterator.from_structure(tf.float32, tf.TensorShape([]))

next_x = iterator.get_next()

init_op = iterator.make_initializer(dataset)

with tf.Session() as sess:
    sess.run(init_op)
    a = sess.run(next_x)
    print(a)
    a = sess.run(next_x)
    print(a)

The trace looks like:

Traceback (most recent call last):
  File “test_iterator_gen.py", line 31, in <module>
    a = sess.run(next_x)
 tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
     [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Iterator)]]

Caused by op 'IteratorGetNext', defined at:
  File "test_iterator_gen.py", line 23, in <module>
    next_x = iterator.get_next()
OutOfRangeError (see above for traceback): End of sequence
     [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Iterator)]]

zephyrus · Accepted Answer · 2017-11-27 20:28:21Z

1

This is caused by incorrect instantiation of a generator.

The error is caused by makeRand() running out of elements to yield. This is fixed by changing it to:

def makeRand():
   while True:
      yield np.random.rand(1)

answered Nov 27, 2017 at 20:28

zephyrus

1,2661 gold badge12 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

TensorFlow Dataset API from_generator End of Sequence Error

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related