3

I'm currently toying around with saving and restoring of variables. For this purpose, I created two scripts. One of them saves a simple graph while the other restores it. Here the test script for saving the graph:

import tensorflow as tf

a = tf.Variable(3.0, name='a')
b = tf.Variable(5.0, name='b')

b = tf.assign_add(b, a)

n_steps = 5

global_step = tf.Variable(0, name='global_step', trainable=False)

saver = tf.train.Saver()

with tf.Session() as sess:

    sess.run(tf.global_variables_initializer())

    for step in range(n_steps):
        print(sess.run(b))

        global_step.assign_add(1).eval()
        print(global_step.eval())

        saver.save(sess, './my_test_model', global_step=global_step)

Basically, I want to run through a loop 5 times and everytime I do this, I add a to b. I also want to keep track of the number of steps via global_step. This works as intended. The output is:

8.0     # value of b
1       # step
11.0
2
14.0
3
17.0
4
20.0
5

Now when restoring the variables, I try to get all three of them. Script is:

import tensorflow as tf

from tensorflow.python.tools.inspect_checkpoint import print_tensors_in_checkpoint_file

# List ALL tensors.
print_tensors_in_checkpoint_file(tf.train.latest_checkpoint('./'), all_tensors=True, tensor_name='')

tf.reset_default_graph()

a = tf.get_variable('a', shape=[])
b = tf.get_variable('b', shape=[])
global_step = tf.get_variable('global_step', shape=[])

saver = tf.train.Saver()

with tf.Session() as sess:

    ckpt = tf.train.latest_checkpoint('./')
    if ckpt:
        print(ckpt)

        saver.restore(sess, ckpt)

    else:
        print('Nothing restored')

    print(a.eval())
    print(b.eval())
    print(global_step.eval())

The output of this is

tensor_name:  a
3.0
tensor_name:  b
20.0
tensor_name:  global_step
5
./my_test_model-5
INFO:tensorflow:Restoring parameters from ./my_test_model-5
3.0
20.0
7e-45

How is it possible that the value for global_step is stored correctly in the checkpoint, but upon evaluation I get this small 7e-45? Also, upon restoring, I seem to be unable to define any additional variables as it states it cannot find the variable in the checkpoint. How can I, for example, define a variable and add it to the b of the restored graph?

Thank you for your help!

1 Answer 1

2

This doesn't appear to be well documented by the TF docs, but you should specify the dtype for the global_step variable.

Incorrect

global_step = tf.get_variable('global_step', shape=[], dtype=tf.float32) results in global_step=7e-5. The type is assumed to be dtf.float32 by default.

Correct

global_step = tf.get_variable('global_step', shape=[], dtype=tf.int32) results in global_step=5

Sign up to request clarification or add additional context in comments.

1 Comment

Can you please open an issue on Github to report this "problem"?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.