Neural network for adding two integer numbers

Question

I want to create a neural network which can add two integer numbers. I have designed it as follows:

question I have really low accuracy of 0.002% . what can i do to increase it?

For creating data:

import numpy as np import random a=[] b=[] c=[]

for i in range(1, 1001): a.append(random.randint(1,999)) b.append(random.randint(1,999)) c.append(a[i-1] + b[i-1])

X = np.array([a,b]).transpose() y = np.array(c).transpose().reshape(-1, 1)
scaling my data :

from sklearn.preprocessing import MinMaxScaler
minmax = MinMaxScaler()
minmax2 = MinMaxScaler()
X = minmax.fit_transform(X)
y = minmax2.fit_transform(y)

The network :


from keras import Sequential
from keras.layers import Dense
from keras.optimizers import SGD

clfa = Sequential()
clfa.add(Dense(input_dim=2, output_dim=2, activation='sigmoid', kernel_initializer='he_uniform'))
clfa.add(Dense(output_dim=2, activation='sigmoid', kernel_initializer='uniform'))
clfa.add(Dense(output_dim=2, activation='sigmoid', kernel_initializer='uniform'))
clfa.add(Dense(output_dim=2, activation='sigmoid', kernel_initializer='uniform'))
clfa.add(Dense(output_dim=1, activation='relu'))

opt = SGD(lr=0.01)
clfa.compile(opt, loss='mean_squared_error', metrics=['acc'])
clfa.fit(X, y, epochs=140)

outputs :

Epoch 133/140
1000/1000 [==============================] - 0s 39us/step - loss: 0.0012 - acc: 0.0020
Epoch 134/140
1000/1000 [==============================] - 0s 40us/step - loss: 0.0012 - acc: 0.0020   
Epoch 135/140
1000/1000 [==============================] - 0s 41us/step - loss: 0.0012 - acc: 0.0020
Epoch 136/140
1000/1000 [==============================] - 0s 40us/step - loss: 0.0012 - acc: 0.0020
Epoch 137/140
1000/1000 [==============================] - 0s 41us/step - loss: 0.0012 - acc: 0.0020
Epoch 138/140
1000/1000 [==============================] - 0s 42us/step - loss: 0.0012 - acc: 0.0020   
Epoch 139/140
1000/1000 [==============================] - 0s 40us/step - loss: 0.0012 - acc: 0.0020   
Epoch 140/140
1000/1000 [==============================] - 0s 42us/step - loss: 0.0012 - acc: 0.0020

That is my code with console outputs..

I have tried every different combinations of optimizers, losses, and activations, plus this data fits perfectly a Linear Regression.

desertnaut · Accepted Answer · 2020-04-29 09:30:36Z

2

Two mistakes, several issues.

The mistakes:

This is a regression problem, so the activation of the last layer should be linear, not relu (leaving it without specifying anything will work, since linear is the default activation in a Keras layer).
Accuracy is meaningless in regression; remove metrics=['acc'] from your model compilation - you should judge the performance of your model only with your loss.

The issues:

We don't use sigmoid activations for the intermediate layers; change all of them to relu.
Remove the kernel_initializer argument, thus leaving the default glorot_uniform, which is the recommended one.
A number of Dense layers each one only with two nodes is not a good idea; try reducing the number of layers and increasing the number of nodes. See here for a simple example network for the iris data.

edited Apr 29, 2020 at 9:30

answered Apr 29, 2020 at 9:08

desertnaut

60.8k32 gold badges155 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Pawandeep Over a year ago

i huge thanks . You are like more than teacher for me. I removed weightinit and used linear activation reeuces layers and now i have accurate predictions and very low loss but accuracy is still 0.002 no matter I have good predictions.

desertnaut Over a year ago

Please notice what I have said in answer: accuracy is meaningless in regression, and you should not use it. Performance assessment should be done exactly based upon loss and quality of predictions.

Pawandeep Over a year ago

Yeah! Thanks i got it

kutschkem · Accepted Answer · 2020-04-29 09:05:58Z

-1

You are trying to fit a linear function, but internally use sigmoid nodes, which map values to a range (0,1). Sigmoid is very useful for classification, but not really for regression if the values are outside (0,1). It could MAYBE work if you restricted your random number to floating point in the interval [0,1]. OR input into your nodes all the bits seperately, and have it learn an adder.

answered Apr 29, 2020 at 9:05

kutschkem

8,2683 gold badges26 silver badges66 bronze badges

3 Comments

Pawandeep Over a year ago

That is why i preprocessed data. Which converted value from 0-1 so my whole dataset in above example is (0,1) values

desertnaut Over a year ago

The values have been normalized before; sigmoid is indeed never used in practice for intermediate layers, but not for the reason you imply here.

kutschkem Over a year ago

@NanduRaj en.wikipedia.org/wiki/Rectifier_(neural_networks) I have to admit I don't have much experience with neural networks, so I guess my knowledge might be a bit outdated. According to wikipedia since 2011 relu is preferred over sigmoid in intermediate layers. A few advantages are listed.

Collectives™ on Stack Overflow

Neural network for adding two integer numbers

2 Answers 2

3 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related