How to connect LSTM layers in Keras, RepeatVector or return_sequence=True?

Question

I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector or return_seq=True. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one?

First model using return_sequence=True:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)

encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)

autoencoder = Model(inputEncoder, outDecoder)

Second model with RepeatVector:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)

firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)

outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)

autoencoder = Model(inputEncoder, outDecoder)

I am curios, what's the reason for RepeatVector(1) in your code. Are you just using it to add a time dimension of 1? But then you follow it up with Reshape((28, 1)) and take it right out... Or did I misunderstand something in your code? — Intelligent Shade of Blue
– Intelligent Shade of Blue, Commented Aug 30, 2020 at 14:34

Dave · Accepted Answer · 2019-10-03 14:58:00Z

71

You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches.

Essentially, return_sequences=True returns all the outputs the encoder observed in the past, while RepeatVector repeats the very last output of the encoder.

edited Oct 3, 2019 at 14:58

Dave

3592 silver badges16 bronze badges

answered Aug 8, 2018 at 23:32

thushv89

11.4k1 gold badge32 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

sariii Over a year ago

Open up my eyes, now I can see better :))

sariii Over a year ago

Again thanks for the insightful answer, do you have any idea where we should use which? Im not saying for 100% but like generally!

thushv89 Over a year ago

Hi @sariii, one example is machine translation. For example if you have a seq2seq model and you don't want to use teacher forcing and want a quick and dirty solution, you can pass in the last state of the encoder RNN (last blue box) using RepeatVector and return_sequences=False. But if you are to compute attention weights for the encoder states, you will need to use return_sequences=True as you need all encoder states to compute attention weights. Hope that makes sense. :)

sariii Over a year ago

@thushv89 Thank you so much for the explanations. Can you please do a favor for me and have a look at this question? stackoverflow.com/questions/56433993/… I have read a lot, even know what is the problem but unable to fix it

thushv89 Over a year ago

@Coderji, I created them. I didn't get it from anywhere :)

|

Collectives™ on Stack Overflow

How to connect LSTM layers in Keras, RepeatVector or return_sequence=True?

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related