if you have a look to the try documentation you can have an example.
Else want you want for your code it :
M = 3;
j = 1;
k = [Inf, 5, 4];
while M>0
try
M = M-1
u = k(j)
if (isinf(u)||isnan(u)), error(''), end;
catch
disp('I catch an error!');
j = j+1
end
end
Because If you never get an error in your code, it will never go in the catch. So by including error('');, it just to say, go execute the statement in the catch.
But you can just modify your code by replacing the error() by the statements into your catch like this :
while M>0
M = M-1
u = k(j)
if (isinf(u)||isnan(u)), j = j+1, end;
end
EDIT
If you take a look in the documentation, you can found this :
% ERROR(MSGSTRUCT) reports the error using fields stored in the scalar
% structure MSGSTRUCT. This structure can contain these fields:
%
% message - Error message string
% identifier - See MESSAGE IDENTIFIERS, below
% stack - Struct similar to the output of the DBSTACK function
%
% If MSGSTRUCT is an empty structure, no action is taken and ERROR
% returns without exiting the program. If you do not specify the
% stack, the ERROR function determines it from the current file and line.
So no action is taken as you can read. And nothing, so catch don't get any informations.