Rescaling input features for Neural Networks (Regression)

Question

In Neural Nets for the regression problem, we rescale the continuous labels consistently with the output activation function, i.e. normalize them if the logistic sigmoid is used, or adjusted normalize them if tanh is used. At the end we can restore original range but renormalizing the output neurons back.

Should we also normalize input features? And how? For example, if hidden activation differs from the output activation? E.g. if hidden activation is TANH and output activation is LOGISTIC, should the input features be normalized to lie in [0,1] or [-1,1] interval?

lejlot · Accepted Answer · 2013-10-12 07:00:11Z

5

The short answer is yes, you should also scale the input values, although reasons behind it are quite different then those for output neurons. Activation function simply makes some output values unreachable (sigmoid can output only values in [0,1], tanh in [-1,1]), while this is not true for the input (all activation functions are defined on the whole R domain). Scaling input is performed in order to speed up convergence (so you don't get to the "flat" part of the activation function), but there are no exact rules. At least three possibilities are widely used:

linear scaling to [0,1]
linear scaling to [-1,1]
normalization to the mean=0 and std=1

Each having its own pros and cons for some specific datasets. As far as I know, the last one has the best statistical properties, but it is still a "rule of the thumb" in context of neural networks.

answered Oct 12, 2013 at 7:00

lejlot

67k9 gold badges138 silver badges168 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Oleg Shirokikh Over a year ago

Thanks. I've also noticed that without rescaling the inputs, there are problems with overflow - exponents become huge. However, scaling solves this this problem.

Collectives™ on Stack Overflow

Rescaling input features for Neural Networks (Regression)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related