Why is GradientTape returning None when I use numpy math

Question

I am trying to understand tensorflow GradientTape calculation for RL loss function. When I call a function using np.math the GradientTape returns None. If I use tf.math in the function it works fine. I have looked at tf-agents like ppo and sac and they are doing exactly(?) what I am trying to do (I have tried at last 50 other versions). What's wrong in the code below? What am I missing?

window 10, python 3.6.8, tensorflow 2.0.0 ref:https://github.com/chagmgang/tf2.0_reinforcement_learning/blob/master/policy/ppo.py

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

def my_loss1(x):
    y=tf.sin(x)
    y=tf.abs(y)
    return y

def my_loss2(x):
    y=np.sin(x)
    y=np.abs(y)
    return y

def main(ver):    
    x = np.linspace(0,10,25)
    dsin_dx=np.cos(x)    
    xx = tf.constant(x)
    with tf.GradientTape() as tape:
        tape.watch(xx)
        if ver==0:
            # my_loss1 with tf math 
            loss1=my_loss1(xx)
        if ver==1:
            #my loss with numpy math
            loss1=my_loss2(np.array(xx))    
            loss1 = tf.convert_to_tensor(loss1, dtype=tf.float64)
        print(loss1)
        loss=tf.reduce_sum(loss1)
        print('loss=',loss)
    grads = tape.gradient(loss, xx)

    fig, ax = plt.subplots(2)
    ax[0].plot(x,loss1,'r')
    print('grads', grads)

    if not grads is None:
        ax[1].plot(x, grads)
        ax[1].plot(x,dsin_dx)
    plt.show()

if __name__ == '__main__':
    main(ver=0)  # This works ok
    main(ver=1)  # This returns grads = None

Ulf Wållgren · Accepted Answer · 2020-02-03 13:36:37Z

1

The problem is that the Gradient tape only records tensors. Numpy variables are not recorded why the gradient can't be calqulated in case ver=1. Loss1 in ver1 looks identical to loss1 in ver=0 but the dependentsy to xx is broken by numpy.
My ref. has this error when calculation get_gaes() and the calculation of the grads is incorrect.

edited Feb 3, 2020 at 13:36

answered Jan 27, 2020 at 22:35

Ulf Wållgren

1712 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jochen Sauter · Accepted Answer · 2023-08-29 14:37:34Z

0

It can be even more malicious: I had the case, that if there is any numpy array somewhere in the code under the gradient tape, there are gradients computed that look quite reasonable, but are actually wrong. No warnings whatsoever were issued by tensorflow. Replacing the numpy array by tf variables fixed it.

answered Aug 29, 2023 at 14:37

Jochen Sauter

511 silver badge2 bronze badges

Collectives™ on Stack Overflow

Why is GradientTape returning None when I use numpy math

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related