Custom training Tensorflow on Sagemaker: WARNING keras.py:603 cannot save layer values during forward pass with tf.function

Question

I am training a custom model with a custom layer using Tensorflow on Sagemaker. It runs smoothly local, no errors, no warnings. When I run it on Sagemaker, it produces many warnings, all the same:

2021-01-22 15:20:26.660 xxxx.xxxx.compute.internal:32 WARNING keras.py:603] cannot save layer values during forward pass with tf.function

I am not sure if this is something to be worried about. Does someone have any ideas about this?

Danny Varod · Accepted Answer · 2021-01-27 13:55:54Z

1

This warning is logged smdebug/tensorflow/keras.py from changeset 9ad5d8251dfa3cab2be494cd49f06678a6a84a4e, line 603 from 2020-12-08.

In changeset da3b4e30b09dda4813c7b6b3bef0e7fb0c8243cf, line 604 from 2020-12-23 the warning was removed, so apparently the authors decided the warning is redundant.

answered Jan 27, 2021 at 13:55

Danny Varod

18.3k5 gold badges76 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Custom training Tensorflow on Sagemaker: WARNING keras.py:603 cannot save layer values during forward pass with tf.function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related