Access lower dimensional encoded data of autoencoder

Question

Here is an autoencoder I’m working on from tutorial:https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/

I’m just learning about autoencoders and I’ve modified the source encode a custom small dataset which consists of:

[0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0]

It seems to work ok, but I’m unsure how to access the lower dimensional embedding values of dimension 2 (set by parameter out_features).

I've added a methods to the Autoencoder class to return the embedding , is this the recommended method of accessing the embedding's ?

Code:

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import MNIST
from torchvision.utils import save_image
import warnings
import os

# import packages
import os
import torch 
import torchvision
import torch.nn as nn
import torchvision.transforms as transforms
import torch.optim as optim
import matplotlib.pyplot as plt
import torch.nn.functional as F

from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.utils import save_image
import numpy as np

# utility functions
def get_device():
    if torch.cuda.is_available():
        device = 'cuda:0'
    else:
        device = 'cpu'
    return device

device = get_device()

features = torch.tensor(np.array([ [0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0] ])).float()
tic_tac_toe_data_loader = torch.utils.data.DataLoader(features, batch_size=1, shuffle=True)

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.fc1 = nn.Linear(in_features=9, out_features=2)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        self.fc1 = nn.Linear(in_features=2, out_features=9)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))


class Autoencoder(nn.Module):

    def __init__(self):
        super(Autoencoder, self).__init__()
        self.fc1 = Encoder()
        self.fc2 = Decoder()

    def forward(self, x):
        return self.fc2(self.fc1(x))

net = Autoencoder()
net.to(device)

NUM_EPOCHS = 50
LEARNING_RATE = 1e-3

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

# image transformations
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

outputs = None

def train(net, trainloader, NUM_EPOCHS):
    train_loss = []
    for epoch in range(NUM_EPOCHS):
        running_loss = 0.0
        for data in trainloader:
            img = data
            img = img.to(device)
            img = img.view(img.size(0), -1)
#             print('img.shape' , img.shape)
            optimizer.zero_grad()

            outputs = net(img)
            loss = criterion(outputs, img)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        loss = running_loss / len(trainloader)
        train_loss.append(loss)

    return train_loss

# train the network
train_loss = train(net, tic_tac_toe_data_loader, NUM_EPOCHS)

I can access the lower dimensional embedding using

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

But is this using the trained weight values for the embedding ? If I call Encoder multiple times with same values:

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))
print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

different results are returned:

tensor([0.5083, 0.5020], grad_fn=<SigmoidBackward>)
tensor([0.4929, 0.6940], grad_fn=<SigmoidBackward>)

Why is this the case ? Is an extra training step being invoked as a result of calling Encoder ?

dumbPy · Accepted Answer · 2020-05-16 09:46:17Z

2

By calling Encoder() you are basically creating a new instance of the encoder everytime and the weights are randomly initialized each time.

Generally, you make one instance of it and train it, save the weights, and infer on it.

Also, for PyTorch, you need not call .forward(), but call the instance directly. Forward is called by it implicitly, including other hook methods if any.

enc = Encoder()
input = torch.from_numpy(np.asarray([0,1,0,1,0,1,0,1,0]).float()
print(enc(input))
print(enc(input))

Training pass happens when you pass the Encode() instance to train function. Calling Encoder() only creates a new object.
Since each object has it's own weights, and the weights are initialized randomly (see xavier and kaiming initialization), you are different outputs. Moving to a single object, you still have to explicitly train it with the train function.

edited May 16, 2020 at 9:46

answered May 16, 2020 at 1:32

dumbPy

1,5981 gold badge9 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

dumbPy Over a year ago

Any closure on this?

blue-sky Over a year ago

thanks! typo in your code I think - storch.from_numpy(np.asarray([0,1,0,1,0,1,0,1,0]).float()) hould be torch.from_numpy(np.asarray([0,1,0,1,0,1,0,1,0])).float() . So each time I call Encoder() a new training pass is invoked ?

dumbPy Over a year ago

Updated answer including for this question

Mabu · Accepted Answer · 2020-05-22 00:57:02Z

Like other responder pointed out when you call Encoder() you generate new instances with randomly initialized weights. Because you are interested in lower dimensional embedding produced by your encoder you need to access the weights of the encoder in your trained net:

trained_encoder = net.fc1

Now that you have your encoder with trained weights the following lines should produces same result:

print(trained_encoder.forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))
print(trained_encoder.forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

As pointed out by others you can further simplify by passing input directly:

test_input = torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()
print(trained_encoder(test_input))

Collectives™ on Stack Overflow

Access lower dimensional encoded data of autoencoder

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related