TensorFlow: Initializing variables multiple times

Question

I am a bit confused by how the following code segment runs.

import tensorflow as tf

x = tf.Variable(0)
init_op = tf.initialize_all_variables()
modify_op = x.assign(5)

with tf.Session() as sess:
    sess.run(init_op)
    print(sess.run(x))
    x += 3
    print(sess.run(x))
    sess.run(init_op) # Trying to initialize x once again to 0
    print(sess.run(x)) # Gives out 3, which leaves me confused.
    print(sess.run(modify_op))
    print(sess.run(x)) # Gives out 8, even more confusing

This is the output:
0
3
3
5
8

Is it that the line x += 3 is not part of the default graph? Or something else is going on? Some help will be appreciated, thanks!

From the look of it it is because you added the x=x+3 line to the graph. To make it clearer : now the old x that you defined has the name variable_0 or something like that and the new tensor that you print is now no longer variable_0 but variable_1=variable_0+3. It explains why you get 3 (0+3) and 8 (5+3). Variable_0 still exists somewhere though and it has not changed if you do print sess.run(graph.get_tensor_by_name("variable_0")) you will get what you expect. To find out what ere the names of your variable just look at your graph as a graph def. — jeandut
– jeandut, Commented Nov 8, 2016 at 7:46

Neil Slater · Accepted Answer · 2016-11-08 08:21:38Z

3

Your x variable is being changed by

x += 3

but not in a way you might expect. The tensorflow library code over-rides +, so that you are effectively swapping the contents x for a new TF tensor (the old one will still be in the graph, just x now points to a new one). Write it out like this:

x = tf.Variable(0) + 3

and it is clearer what is going on. Also, insert some print statements . . .

x = tf.Variable(0)
print(x)
# <tensorflow.python.ops.variables.Variable object at 0x1018f5d68>

x += 3
print(x)
# Tensor("add:0", shape=(), dtype=int32)

If the contents of x are important to you, then avoid re-assigning to x if you want to track/display x using the variable name later. Alternatively, you can always name the tensor and fetch it direct from the graph if you don't have a convenient Python variable pointing at it. What's important is getting used to the separation between TF variables and Python variables.

Actually seeing the TF variable being assigned and re-set as you are trying to do, needs to use a TF assignment operator:

import tensorflow as tf

x = tf.Variable( 0 )

with tf.Session() as session:
    session.run( tf.initialize_all_variables() )
    print( x.eval() )

    session.run( x.assign( x + 3 ) )
    print( x.eval() )

    session.run( tf.initialize_all_variables() )
    print( x.eval() )

This outputs:

0
3
0

as you were expecting.

edited Nov 8, 2016 at 8:21

answered Nov 8, 2016 at 7:47

Neil Slater

27.4k6 gold badges80 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

stark Over a year ago

Thanks, that clears it up. So, I should stick to using the tf.add, tf.assign and tf.assign_add family, for the changes to be reflected in the graph.

Neil Slater Over a year ago

@stark: You don't have to, and that would not save you here. What you should do is avoid re-assigning to x if you want to track/display x using the variable name later. Alternatively, you can always name the tensor and fetch it direct from the graph if you don't have a convenient Python variable pointing at it. What's important is getting used to the separation between TF variables and Python variables.

stark Over a year ago

Just realized that. I'll play around with it a bit more, thanks for the tips!

Neil Slater Over a year ago

@stark: I added more code to the answer that shows the call to initialise_all_variables() re-setting x as you were trying to test before. Note how I do the assignment to x - using a tf operation . . .

stark Over a year ago

That is what I did, after you pointed out that python variables and TF managed variables on the graph are different. Thanks for the detailed explanation.

Collectives™ on Stack Overflow

TensorFlow: Initializing variables multiple times

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related