Normally for a recursive function to work it needs to have a base case where the recursion stops. It also needs the recursive step to be simpler than the original problem, thus less work, such that eventually it will hit the base case.
So looking at your code:
function factorial(n) {
if (n === 0) {
return 0;
}else {
return (n * factorial((n-1)*(n-1))); // problem is here
}
}
It has a base case where it returns 0 when n is zero. Thus factorial(0) // ==> 0. Now lets say we try it with 3.
factorial(3); // ==
3 * factorial((3-1)*(3-1)); // ==
3 * factorial(4); // ==
3 * 4 * factorial((4-1)*(4-1)); // ==
3 * 4 * factorial(9); // ==
The requirement that the recursion is a simpler problem. factorial(4) is not simpler than factorial(3) so this recursion will never ever hit the base case.
If you want to implement factorial the base case should return 1 and not 0 and the recursion is always one less then the current value. factorial(2) is a simpler problem than factorial(3) since it will hot the base case such that the result is 3 * 2 * 1 * 1.
0! === 1.