Dealing with overflow in Java without using BigInteger

To answer your first question (why did you get zero), the values of fact() as computed by modular arithmetic were such that you hit a result with all 64 bits zero! Change your fact code to this:

public static long fact(int n) {
    long rs = 1;
    if( n <2) return 1;
    for (int i=2; i<=n; i++) {
        rs *=i;
        System.out.println(rs);
    }
    return rs;
}

Take a look at the outputs! They are very interesting.

Now onto the second question....

It looks like you want to give exact integer (er, long) answers for values of n and r that fit, and throw an exception if they do not. This is a fair exercise.

To do this properly you should not use factorial at all. The trick is to recognize that C(n,r) can be computed incrementally by adding terms. This can be done using recursion with memoization, or by the multiplicative formula mentioned by Stefan Kendall.

As you accumulate the results into a long variable that you will use for your answer, check the value after each addition to see if it goes negative. When it does, throw an exception. If it stays positive, you can safely return your accumulated result as your answer.

To see why this works consider Pascal's triangle

1
1  1
1  2   1
1  3   3   1
1  4   6   4   1
1  5  10  10   5  1
1  6  15  20  15  6  1

which is generated like so:

C(0,0) = 1 (base case)
C(1,0) = 1 (base case)
C(1,1) = 1 (base case) 
C(2,0) = 1 (base case) 
C(2,1) = C(1,0) + C(1,1) = 2
C(2,2) = 1 (base case)
C(3,0) = 1 (base case)
C(3,1) = C(2,0) + C(2,1) = 3
C(3,2) = C(2,1) + C(2,2) = 3
...

When computing the value of C(n,r) using memoization, store the results of recursive invocations as you encounter them in a suitable structure such as an array or hashmap. Each value is the sum of two smaller numbers. The numbers start small and are always positive. Whenever you compute a new value (let's call it a subterm) you are adding smaller positive numbers. Recall from your computer organization class that whenever you add two modular positive numbers, there is an overflow if and only if the sum is negative. It only takes one overflow in the whole process for you to know that the C(n,r) you are looking for is too large.

This line of argument could be turned into a nice inductive proof, but that might be for another assignment, and perhaps another StackExchange site.

ADDENDUM

Here is a complete application you can run. (I haven't figured out how to get Java to run on codepad and ideone).

/**
 * A demo showing how to do combinations using recursion and memoization, while detecting
 * results that cannot fit in 64 bits.
 */
public class CombinationExample {

    /**
     * Returns the number of combinatios of r things out of n total.
     */
    public static long combi(int n, int r) {
        long[][] cache = new long[n + 1][n + 1];
        if (n < 0 || r > n) {
            throw new IllegalArgumentException("Nonsense args");
        }
        return c(n, r, cache);
    }

    /**
     * Recursive helper for combi.
     */
    private static long c(int n, int r, long[][] cache) {
        if (r == 0 || r == n) {
            return cache[n][r] = 1;
        } else if (cache[n][r] != 0) {
            return cache[n][r];
        } else {
            cache[n][r] = c(n-1, r-1, cache) + c(n-1, r, cache);
            if (cache[n][r] < 0) {
                throw new RuntimeException("Woops too big");
            }
            return cache[n][r];
        }
    }

    /**
     * Prints out a few example invocations.
     */
    public static void main(String[] args) {
        String[] data = ("0,0,3,1,4,4,5,2,10,0,10,10,10,4,9,7,70,8,295,100," +
                "34,88,-2,7,9,-1,90,0,90,1,90,2,90,3,90,8,90,24").split(",");
        for (int i = 0; i < data.length; i += 2) {
            int n = Integer.valueOf(data[i]);
            int r = Integer.valueOf(data[i + 1]);
            System.out.printf("C(%d,%d) = ", n, r);
            try {
                System.out.println(combi(n, r));
            } catch (Exception e) {
                System.out.println(e.getMessage());
            }
        }
    }
}

Hope it is useful. It's just a quick hack so you might want to clean it up a little.... Also note that a good solution would use proper unit testing, although this code does give nice output.