TF-Keras - Dataset.from_generator for multi-input functional API model

Question

I have a generator that yields three variables. The first two variables are the two inputs to a two-input Keras model (functional API). I am using TF-Dataset to feed my model. Code is as follows:


train_dataset = tf.data.Dataset.from_generator(generator=make_generator_train,
                                                   args=[train_x_paths, train_y_int],
                                                   output_types=(tf.tuple((tf.float16, tf.float16)), tf.int8),
                                                   output_shapes=(tf.TensorShape([2]),
                                                                  tf.TensorShape([1]))).batch(batch_size=batch_size)

I get a TypeError with this:

TypeError: If shallow structure is a sequence, input must also be a sequence. Input has type: <class 'tensorflow.python.framework.tensor_shape.TensorShape'>.

tornikeo · Accepted Answer · 2020-08-22 09:40:20Z

1

Try it like this:

train_dataset = tf.data.Dataset.from_generator(
        generator=make_generator_train,
       args=[train_x_paths, train_y_int],
       output_types=(tf.float16, tf.int8)
).batch(batch_size=batch_size)

Most of the time you don't need to specify the output_shapes. It is decided in the runtime. Additionally, you only need to specify the overall dtype of the output tensor in the output_types. Not a dtype for each individual tensor dimension.

answered Aug 22, 2020 at 9:40

tornikeo

9828 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sravan953 Over a year ago

Skipping the output_shapes parameter did not fix it, I had to fix my generator instead. Thanks!

sravan953 · Accepted Answer · 2020-08-24 13:42:54Z

1

Solution: the generator should yield a dictionary for the inputs, and the output as is.

answered Aug 24, 2020 at 13:42

sravan953

2911 gold badge4 silver badges14 bronze badges

Collectives™ on Stack Overflow

TF-Keras - Dataset.from_generator for multi-input functional API model

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related