Setting constraints for parameters in pytorch

Question

I am trying to set some constraints for weight parameters in PyTorch, e.g. the sum of every row of the weight matrix be exactly one for a fully connected layer:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(28*28, 10*10)
        self.layer2 = nn.Linear(10*10, 5*5)
        self.layer3 = nn.Linear(5*5, 10)

    def forward(self, x):
        x=torch.sigmoid(self.layer1(x))
        x=torch.sigmoid(self.layer2(x))
        x=torch.sigmoid(self.layer3(x))

model=Net()

The constraint for this example network would be:

torch.sum(model.linear1.weight,0)==1
torch.sum(model.linear2.weight,0)==1
torch.sum(model.linear3.weight,0)==1

A commonly used method to set a constraint, clamp, is used to set constraints for every element, but in this case, I would be setting a constraint for every row, instead of any particular element of the weight matrix. Are there any ways to implement this kind of constraint?

Please define your question better in terms of inputs and required outputs. As it is now, it is not answerable. — Gulzar
– Gulzar, Commented May 23, 2021 at 9:03
why do you want this constraint for weights in the first place? Is it a experiment? — R Nanthak
– R Nanthak, Commented May 23, 2021 at 9:32
@RNanthak Some more complex constraints are required for the experiment, but I stumble for even the most simple cases, sorry! — t-smart
– t-smart, Commented May 23, 2021 at 12:14

Gulzar · Accepted Answer · 2021-05-23 13:21:25Z

1

A way I can think about is for example to normalize the vector of your choosing by its norm, which will give its direction, with size 1.

w0 = model.linear1.weight[0, :]
w0_hat = w0 / torch.linalg.norm(w0) # direction of w0, norm=1

I don't really see a way of doing this for the .sum, but I also don't see why one would want to.

Using L1 norm would do that for the .sum if you can guarantee the weights are all non-negative. Probably this isn't what you want.

If you insist of normalizing such that the .sum will equal to 1, you have to define the normalization yourself, because there is no single algorithm to decide which weight index gets changed, and by how much.

answered May 23, 2021 at 13:21

Gulzar

28.8k42 gold badges159 silver badges260 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

t-smart Over a year ago

The first implementation is fine! The .sum was just for clarification purposes and is indeed unnecessary.

Collectives™ on Stack Overflow

Setting constraints for parameters in pytorch

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related