tmult_ok(x, y) fails anytime x*y in int p = x*y; overflows as that is undefined behavior (UB).
It also fails a corner case like tmult_ok(INT_MIN, -1) for the same reason.
It does not portably "work".
An alternative (and others for /,-,+) which does not depend on 2's complement. Notice this returns the opposite of tmult_ok().
int is_undefined_mult1(int a, int b) {
if (a > 0) {
if (b > 0) {
return a > INT_MAX / b; // a positive, b positive
}
return b < INT_MIN / a; // a positive, b not positive
}
if (b > 0) {
return a < INT_MIN / b; // a not positive, b positive
}
return a != 0 && b < INT_MAX / a; // a not positive, b not positive
}
How do I ensure that p != x*y when there is an overflow?
Portable code cannot. With signed integer math in C, overflow is UB. Code should detect a potential overflow without first performing the multiplication. @Quentin @Eugene Sh.
how do I prove its correctness in all the cases?
Form a reference test with uses 2x wide math. If int is 32-bit, compare tmult_ok() to a multiplication using 64-bit math and see if the product in in range. @rici
int tmult_ok_ll(int x, int y) {
long long prod = x;
prod *= y;
return (prod >= INT_MIN && prod <= INT_MAX);
}
Trying all combinations is a brute force approach - likely too long for 32-bit int.
Try a subset of all combinations, for each x,y,try INT_MIN, INT_MIN-1, INT_MIN-2, -2,-1, 0, 1, 2, , INT_MAX-1, INT_MAX. (10*10 tests)
Also a subset of all combinations, for each values +/- within 2 of sqrt(INT_MAX). (10*10 tests)
Also a few million random values in the int range would be prudent.
This may not be sufficient, yet if code passes this, there are very few corner cases left - which are very dependent on your source code.
See also @Eric Postpischil