Saving and loading a keras model

Question

I have trained a model which has among others 2 embeddings which have initial weights pre-trained (using gensim). Both embeddings have trainable=True meaning that the weights are updated in the training. The model is something like the following:

def init_embedding(emb_matrix, name):
    embedding_layer = Embedding(
        emb_matrix.shape[0],
        emb_matrix.shape[1],
        weights=[emb_matrix],
        trainable=True,
        name=name)
    return embedding_layer

embedding_layer1 = init_embedding(gensimWord2vecMatrix, "emb_1")
embedding_layer2 = init_embedding(gensimWord2vecMatrix2, "emb_2")


sequence_input_1 = Input(shape=(10,), dtype='int32', name="input_1")
sequence_input_2 = Input(shape=(1,), dtype='int32', name="input_2")
sequence_input_3 = Input(shape=(100,), dtype='int32', name="input_3")


embedded_sequences_1 = embedding_layer1(sequence_input_1)
embedded_sequences_2 = embedded_layer2(sequence_input_1)
embedded_sequences_3 = embedded_layer2(sequence_input_3)

conv_step1 = Convolution1D(filters=1000,kernel_size=5,activation=activation_fun,name="conv_layer_",padding="valid")(embedded_sequences_1)
        .
        .

z = Dense(ashape, activation="linear", name="predicted_description")(conv_step10)
loss = a_loss([z, embedded_sequences_2, embedded_sequences_3])# a loss function
model = Model(
    inputs=[sequence_input_1, sequence_input_2, sequence_input_],
    outputs=[loss])
    model.compile(loss=mean_loss, optimizer=Adam())

and what I am trying to do is to save/load the model. The first that I tried was to:

model.save("./test_model.h5")

But after when I use:

from keras.models import load_model
model = load_model("./test_model.h5")

I got an error of : ValueError: Missing layer: x

Searching through the internet a solution is to save only the weights and define again the model (without the compile) and assign the saved weights. Something like:

model.save_weights("./model_weights.h5")

and again difining the model without the model.compile in the end:

embedding_layer1 = init_embedding(gensimWord2vecMatrix, "emb_1")
embedding_layer2 = init_embedding(gensimWord2vecMatrix2, "emb_2")
         .
         .
         .
model = Model(...
model.load_weights("./test_model.h5")

which seems to work fine and I can use the predict function. My question is what happens with the trainable embeddings using this methodology. Are the predict functions using the trained weights from the model or it using the weights that are initialized from the gensim model? because to use the load_weights as shown above I am recreating the embeddings from the beginning.

Any idea? I hope my question was clear. Any comment is useful for restructuring the question if its not understandable.

It would be a good idea to post exactly the code you're using to define your model. Aren't the embeddings a part of it? — Daniel Möller
– Daniel Möller, Commented Sep 21, 2017 at 23:21

Daniel Möller · Accepted Answer · 2017-09-22 07:27:14Z

1

If you create the embeddings first and later you load weights, the loaded weights will prevail (as long as you never call init embedding after that).

You can also make a manual check.

Create the model, get each embedding layer weights:

w1 = model.get_layer('embeddingName1').get_weights()
#same for 2 and 3

Load the weights and get them again from the layers:

newW1 = model.get_layer('embeddingName1').get_weights()

Compare them:

for w,newW in zip(w1,newW1):
    print((w==newW).all()) #if false, weights have changed with loading   

#same for 2 and 3.

answered Sep 22, 2017 at 7:27

Daniel Möller

86.8k24 gold badges202 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mpizos Dimitris Over a year ago

So in the second run (when loading the model) it doesn't matter if I initialize randomly the embeddings or initilizing with the pre-trained weights because they will be overwritten anyway from load_weights. Right?

Daniel Möller Over a year ago

Yes :) -- Loading weights will overwrite everything.

Collectives™ on Stack Overflow

Saving and loading a keras model

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related