How to create custom objective function in Keras?

Question

There are many objective functions in Keras here.

But how can you create your own objective function, I tried to create a very basic objective function but it gives an error and I there is no way to know the size of the parameters passed to the function at run time.

def loss(y_true,y_pred):
    loss = T.vector('float64')
    for i in range(1):
        flag = True
        for j in range(y_true.ndim):
            if(y_true[i][j] == y_pred[i][j]):
                flag = False
        if(flag):
            loss = loss + 1.0
    loss /= y_true.shape[0]
    print loss.type
    print y_true.shape[0]
    return loss

I am getting 2 contradicting errors,

model.compile(loss=loss, optimizer=ada)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/models.py", line 75, in compile
    updates = self.optimizer.get_updates(self.params, self.regularizers, self.constraints, train_loss)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/optimizers.py", line 113, in get_updates
    grads = self.get_gradients(cost, params, regularizers)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/optimizers.py", line 23, in get_gradients
    grads = T.grad(cost, params)
  File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 432, in grad
    raise TypeError("cost must be a scalar.")
TypeError: cost must be a scalar.

It says cost or loss returned in the function must be a scalar but if I change the line 2 from loss = T.vector('float64')
to
loss = T.scalar('float64')

it shows this error

 model.compile(loss=loss, optimizer=ada)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/models.py", line 75, in compile
    updates = self.optimizer.get_updates(self.params, self.regularizers, self.constraints, train_loss)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/optimizers.py", line 113, in get_updates
    grads = self.get_gradients(cost, params, regularizers)
  File "/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg/keras/optimizers.py", line 23, in get_gradients
    grads = T.grad(cost, params)
  File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 529, in grad
    handle_disconnected(elem)
  File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 516, in handle_disconnected
    raise DisconnectedInputError(message)
theano.gradient.DisconnectedInputError: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: <TensorType(float64, matrix)>

Your loss should be a Theano function of y_true and y_pred, i.e. it has to be expressed in term of tensor operations on these parameters. — Mikael Rousson
– Mikael Rousson, Commented Nov 23, 2015 at 21:17

Boern · Accepted Answer · 2017-12-05 06:22:43Z

Here is my small snippet to write new loss functions and test them before using:

import numpy as np

from keras import backend as K

_EPSILON = K.epsilon()

def _loss_tensor(y_true, y_pred):
    y_pred = K.clip(y_pred, _EPSILON, 1.0-_EPSILON)
    out = -(y_true * K.log(y_pred) + (1.0 - y_true) * K.log(1.0 - y_pred))
    return K.mean(out, axis=-1)

def _loss_np(y_true, y_pred):
    y_pred = np.clip(y_pred, _EPSILON, 1.0-_EPSILON)
    out = -(y_true * np.log(y_pred) + (1.0 - y_true) * np.log(1.0 - y_pred))
    return np.mean(out, axis=-1)

def check_loss(_shape):
    if _shape == '2d':
        shape = (6, 7)
    elif _shape == '3d':
        shape = (5, 6, 7)
    elif _shape == '4d':
        shape = (8, 5, 6, 7)
    elif _shape == '5d':
        shape = (9, 8, 5, 6, 7)

    y_a = np.random.random(shape)
    y_b = np.random.random(shape)

    out1 = K.eval(_loss_tensor(K.variable(y_a), K.variable(y_b)))
    out2 = _loss_np(y_a, y_b)

    assert out1.shape == out2.shape
    assert out1.shape == shape[:-1]
    print np.linalg.norm(out1)
    print np.linalg.norm(out2)
    print np.linalg.norm(out1-out2)


def test_loss():
    shape_list = ['2d', '3d', '4d', '5d']
    for _shape in shape_list:
        check_loss(_shape)
        print '======================'

if __name__ == '__main__':
    test_loss()

Here as you can see I am testing the binary_crossentropy loss, and have 2 separate losses defined, one numpy version (_loss_np) another tensor version (_loss_tensor) [Note: if you just use the keras functions then it will work with both Theano and Tensorflow... but if you are depending on one of them you can also reference them by K.theano.tensor.function or K.tf.function]

Later I am comparing the output shapes and the L2 norm of the outputs (which should be almost equal) and the L2 norm of the difference (which should be towards 0)

Once you are satisfied that your loss function is working properly you can use it as:

model.compile(loss=_loss_tensor, optimizer=sgd)

You rule! +1 for test driven development - there is nothing better for machine learning.

razimbres · Accepted Answer · 2018-01-09 18:15:57Z

4

(Answer Fixed) A simple way to do it is calling Keras backend:

import keras.backend as K

def custom_loss(y_true,y_pred):
    return K.mean((y_true - y_pred)**2)

Then:

model.compile(loss=custom_loss, optimizer=sgd,metrics = ['accuracy'])

that equals

model.compile(loss='mean_squared_error', optimizer=sgd,metrics = ['accuracy'])

edited Jan 9, 2018 at 18:15

answered Sep 30, 2016 at 10:29

razimbres

5,0955 gold badges29 silver badges55 bronze badges

2 Comments

Kev1n91 Over a year ago

that is just the usual loss, not a custom loss function

Yan King Yin Over a year ago

agree with @Kev1n91, I've never seen a non-trivial example of custom loss function that works

Collectives™ on Stack Overflow

How to create custom objective function in Keras?

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related