Cannot assign value to tensor variable loaded from graph

Question

I've trained a model and saved it. Now, I'm trying to see how weights perturbations might affect its accuracy, so I need to modify the values saved on my weights variables essentially adding some noise to it. The problem is that I can't assign a value to them after I've loaded them. I'm using tensorflow version 1.2.1., to train and load the model. Here is my code:

import tensorflow as tf
tf.reset_default_graph()
sess = tf.InteractiveSession()
saver = tf.train.import_meta_graph('/scratch/pedro/TFModels/Checks_and_Logs/20170803_215828/beta_model-1.meta')
print("Graph restored")
saver.restore(sess, tf.train.latest_checkpoint('/scratch/pedro/TFModels/Checks_and_Logs/20170803_215828/'))
print("Model restored")
tf.global_variables() #prints the list of variables in the graph

This produces the following output:

[<tf.Variable 'FF_NN/Model/hidden_layer_1/weights/Variable:0' shape=(3960, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/hidden_layer_1/bias/bias_hidden_layer_1:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/hidden_layer_2/weights/Variable:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/hidden_layer_2/bias/bias_hidden_layer_2:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/hidden_layer_3/weights/Variable:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/hidden_layer_3/bias/bias_hidden_layer_3:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/output_layer/weights/Variable:0' shape=(256, 5) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/output_layer/bias/bias_output_layer:0' shape=(5,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/Variable:0' shape=() dtype=int32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/beta1_power:0' shape=() dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/beta2_power:0' shape=() dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_1/weights/Variable/Adam:0' shape=(3960, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_1/weights/Variable/Adam_1:0' shape=(3960, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_1/bias/bias_hidden_layer_1/Adam:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_1/bias/bias_hidden_layer_1/Adam_1:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_2/weights/Variable/Adam:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_2/weights/Variable/Adam_1:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_2/bias/bias_hidden_layer_2/Adam:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_2/bias/bias_hidden_layer_2/Adam_1:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_3/weights/Variable/Adam:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_3/weights/Variable/Adam_1:0' shape=(256, 256) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_3/bias/bias_hidden_layer_3/Adam:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/hidden_layer_3/bias/bias_hidden_layer_3/Adam_1:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/output_layer/weights/Variable/Adam:0' shape=(256, 5) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/output_layer/weights/Variable/Adam_1:0' shape=(256, 5) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/output_layer/bias/bias_output_layer/Adam:0' shape=(5,) dtype=float32_ref>,
 <tf.Variable 'FF_NN/Model/Training/Optimizer/FF_NN/Model/output_layer/bias/bias_output_layer/Adam_1:0' shape=(5,) dtype=float32_ref>]

So, I've been trying to modify the first one ( FF_NN/Model/hidden_layer_1/weights/Variable:0) but that gives me an error:

x = data_train[:batch_size]
y = data_train_labels[:batch_size]
graph = tf.get_default_graph()
data_train_tensor = graph.get_tensor_by_name("Train_Dataset:0")
data_train_labels_onehot = graph.get_tensor_by_name("Train_Labels:0")
acc_te = graph.get_tensor_by_name("Test_Data_Accuracy/Mean:0")
acc_tr = graph.get_tensor_by_name("Train_Data_Accuracy/Mean:0")


w1 = graph.get_tensor_by_name("FF_NN/Model/hidden_layer_1/weights/Variable:0")
print('w1:\n', w1.eval())
training_acc, test_acc = sess.run([acc_tr, acc_te], feed_dict={data_train_tensor: x, data_train_labels_onehot: y})
print(test_acc)

w1 = w1 + 50
print('w1:\n', w1.eval())
sess.run(w1.assign(w1))
training_acc, test_acc, _ = sess.run([acc_tr, acc_te, w1], feed_dict={data_train_tensor: x, data_train_labels_onehot: y})
print(test_acc)

This gives me an error in the assign operation:

w1:
 [[-0.0531723   0.73768502  0.14098917 ...,  1.67111528  0.2495033
   0.20415793]
 [ 1.20964873 -0.99254322 -3.01407313 ...,  0.40427083  0.33289135
   0.2326804 ]
 [ 0.70157909 -1.61257529 -0.59762233 ...,  0.20860809 -0.02733657
   1.57942903]
 ..., 
 [ 1.23854971 -2.28062844 -1.01647282 ...,  1.18426156  0.65342903
  -0.45519635]
 [ 1.02164841 -0.11143603  1.71673298 ..., -0.85511237  1.15535712
   0.50917912]
 [-2.52524352 -0.04488864  0.66239733 ..., -0.45516238 -0.76003599
  -1.2073245 ]]
0.242335
w1:
 [[ 49.94682693  50.73768616  50.1409874  ...,  51.67111588  50.24950409
   50.20415878]
 [ 51.20964813  49.00745773  46.98592758 ...,  50.40427017  50.33288956
   50.23268127]
 [ 50.70158005  48.38742447  49.40237808 ...,  50.20860672  49.97266388
   51.57942963]
 ..., 
 [ 51.23854828  47.7193718   48.98352814 ...,  51.18426132  50.65342712
   49.54480362]
 [ 51.02164841  49.88856506  51.71673203 ...,  49.14488602  51.15535736
   50.50917816]
 [ 47.47475815  49.95511246  50.66239548 ...,  49.54483795  49.23996353
   48.79267502]]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-34-da5d05216392> in <module>()
     16 w1 = w1 +50
     17 print('w1:\n', w1.eval())
---> 18 sess.run(w1.assign(w1))
     19 #print('w1:\n', w1.eval())
     20 training_acc, test_acc, _ = sess.run([acc_tr, acc_te, w1], feed_dict={data_train_tensor: x, data_train_labels_onehot: y})

AttributeError: 'Tensor' object has no attribute 'assign'

All the similar questions point out to the fact that w1 should be a tf.Variable type and that seems to be the case here, according to the output of tf.global_variables().

w1 used to point to tf.Variable, but then you made it point to a tf.add tensor which has no assign method — Yaroslav Bulatov
– Yaroslav Bulatov, Commented Aug 9, 2017 at 17:25
I understand what you said but I'm still unsure how to do this. Even if I do w2 = w1 + 50 and then w1.assign(w2).eval() I get the same error. — Pedro Silva
– Pedro Silva, Commented Aug 9, 2017 at 17:30
ah, sorry, didn't read it carefully enough...doing get_tensor_by_name gives you a Tensor object. A tf.Variable contains that tensor object and some other stuff (like assign method). You should try using collections to get the Python variable object restored (ie, like in tf.get_collection('train_op')[0]) — Yaroslav Bulatov
– Yaroslav Bulatov, Commented Aug 9, 2017 at 17:33
Thanks, @YaroslavBulatov . tf.get_collection('train_op') returned an empty list but that led me to take a look at the other options in tf.GraphKeys and I managed to get it using tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)[0]. Interestingly enough tf.GraphKeys.WEIGHTS was an empty list. It seems a bit hacky (maybe it's my fault for not defining proper scopes) but I think it solves my problem. If you add it as an answer I'll mark it right. — Pedro Silva
– Pedro Silva, Commented Aug 9, 2017 at 18:18

Ishant Mrinal · Accepted Answer · 2017-08-09 18:34:18Z

3

following code would work. best way to use get_variable

w1 = tf.get_variable("FF_NN/Model/hidden_layer_1/weights/Variable:0")
sess.run(tf.assign(w1, w1+50))

FOR now this step won't work, this is a bug in tensorflow https://github.com/tensorflow/tensorflow/issues/1325

a working solution:

w1 = [v for v in tf.global_variables() if v.name=="FF_NN/Model/hidden_layer_1/weights/Variable:0"][0]
sess.run(tf.assign(w1, w1+50))

edited Aug 9, 2017 at 18:34

answered Aug 9, 2017 at 17:26

Ishant Mrinal

4,9183 gold badges31 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Pedro Silva Over a year ago

I'm unable to read w1 with tf.get_variable(). First I got the error that there wasn't a shape set, so I set the same shape as the original. Then I got this error ValueError: 'FF_NN/Model/hidden_layer_1/weights/Variable:0' is not a valid scope name I removed the :0 and then got the following error:

ValueError: Variable FF_NN/Model/hidden_layer_1/weights/Variable already exists, disallowed. Did you mean to set reuse=True in VarScope?

I don't see a reuse=True option in tf.get_variable though.

Pedro Silva Over a year ago

So that should indeed do what I want. Thanks for pointing out that this is currently a bug, makes me less paranoic.

Yaroslav Bulatov · Accepted Answer · 2017-08-09 18:30:27Z

-1

You need to get underlying variable object using tf.get_variable or tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)[0]

Ishant brings up that get_variable has currently a bug for scoped variables so until it's fixed you need to use get_collection

answered Aug 9, 2017 at 18:30

Yaroslav Bulatov

58.1k23 gold badges147 silver badges210 bronze badges

Collectives™ on Stack Overflow

Cannot assign value to tensor variable loaded from graph

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related