Since x and y have the same length, your diagonal matrix D must be a square matrix of size n x n, with n equal to two times the length of your x or y vectors. The reason why you need to multiply the length by two is because the operation [x,y] concatenates the arrays horizontally thus duplicating one of the dimensions.
In this example D is the Identity matrix. See eye for more information.
x = linspace(-10,10); % x is 1x100
y = linspace(-10,10); % y is 1x100
[X,Y] = meshgrid(x,y); % X is 100x100 and Y is 100x100
D = eye(2*numel(x)); % D is 2*100x2*100 = 200x200
f = @(x,y) [x,y]*D*[x,y].'; % [X,Y] is 100x200 and [X,Y].' is 200x100
contour (X,Y,f(X,Y))
If you want D to be a random diagonal matrix, you can accomplish this combining diag with one of the Random Number Generation functions available, like for example randn.
On the previous example, replace D with the following instruction:
D = diag(randn(1,2*numel(x)));
You can also give the coefficients you choose to the diagonal matrix. To do so, you will need to create the vector of coefficients manually, making sure that it has the adequate length, so that it satisfies the conditions explained at the beginning of this post.
Try now replacing D with the following instructions:
v = 1:2*numel(x); % vector of coefficients: v = [1 2 ... 200]
D = diag(v);
[x,y]is 1 by 2". This is not true, and is the root cause of your problem.XandYare both 100 by 100. So, when you input them tofand there do effectively[X, Y], the result is a 100 by 200 matrix.