Pyth, 16 bytes
!sm-sM._+d_1U2+C
Try it online: Demonstration or Test Suite
Explanation:
!sm-sM._+d_1U2+CQQ two implicit Qs (=input matrix) at the end
+CQQ zip Q and connect it with Q (=list of columns and rows)
m map each column/row d to:
+d_1 append -1 to d
._ compute all prefixes of ^
sM compute the sums of the prefixes
- U2 remove zeros and twosones
a column/row is correct, if this gives an empty list
s connect up all resulting lists
! check, if this result is empty