There are several problems with what you're doing.
For starters, attempting to call the function recursively (function(n+function(n-1))) will result in a call stack size exceeded error because you're adding the sum to each argument passed to the recursive call. You probably want something closer to (n + function(n-1)).
However, this is still problematic, because you can't call an anonymous function recursively (except by using arguments.callee, which is disallowed in strict mode).
To do what it appears you're trying to do without giving a name to your function expression, you could do something like:
console.log(function(n) {
if (n == 0) {
return 1;
}
else {
return n + arguments.callee(n-1);
}
}(8))
However, you could also provide a name for your function expression to refer to in recursive calls, which will work in strict mode as well:
console.log(function foo(n) {
if (n == 0) {
return 1;
}
else {
return n + foo(n-1);
}
}(8))
edited: In the base case (n == 0) you probably want to return 0 instead of 1 if your goal is to get the sum of all whole numbers from 0 to n.