8

I am trying to create my first ensemble models in keras. I have 3 input values and a single output value in my dataset.

from keras.optimizers import SGD,Adam
from keras.layers import Dense,Merge
from keras.models import Sequential

model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation='relu'))
model1.add(Dense(2, activation='relu'))
model1.add(Dense(2, activation='tanh'))
model1.compile(loss='mse', optimizer='Adam', metrics=['accuracy'])

model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation='linear'))
model2.add(Dense(4, activation='tanh'))
model2.add(Dense(3, activation='tanh'))
model2.compile(loss='mse', optimizer='SGD', metrics=['accuracy'])

model3 = Sequential()
model3.add(Merge([model1, model2], mode = 'concat'))
model3.add(Dense(1, activation='sigmoid'))
model3.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])

model3.input_shape

The ensemble model(model3) compiles without any error but while fitting the model I have to pass the same input two times model3.fit([X,X],y). Which I think is an unnecessary step and instead of passing input twice I want to have a common input nodes for my ensemble model. How can I do it?

Improve this question

3 Answers 3

Reset to default
7

Keras functional API seems to be a better fit for your use case, as it allows more flexibility in the computation graph. e.g.:

from keras.layers import concatenate
from keras.models import Model
from keras.layers import Input, Merge
from keras.layers.core import Dense
from keras.layers.merge import concatenate

# a single input layer
inputs = Input(shape=(3,))

# model 1
x1 = Dense(3, activation='relu')(inputs)
x1 = Dense(2, activation='relu')(x1)
x1 = Dense(2, activation='tanh')(x1)

# model 2 
x2 = Dense(3, activation='linear')(inputs)
x2 = Dense(4, activation='tanh')(x2)
x2 = Dense(3, activation='tanh')(x2)

# merging models
x3 = concatenate([x1, x2])

# output layer
predictions = Dense(1, activation='sigmoid')(x3)

# generate a model from the layers above
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Always a good idea to verify it looks as you expect it to 
# model.summary()

data = [[1,2,3], [1,1,3], [7,8,9], [5,8,10]]
labels = [0,0,1,1]

# The resulting model can be fit with a single input:
model.fit(data, labels, epochs=50)

Notes:

  • There might be slight differences in the API between Keras versions (pre- and post- version 2)
  • The example above specifies different optimizer and loss function for each of the models. However, since fit() is being called only once (on model3), the same settings - those of model3 - will apply to the entire model. In order to have different settings when training the sub-models, they will have to be fit() separately - see comment by @Daniel.

EDIT: updated notes based on comments

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

The compilation (optimizer and loss) of a model is only taken into account when you use fit for that specific model. If you are using fit in model3, only the compilation for model3 will take effect. --- There is no need at all to compile model1 and model2, unless you're going to train them separately (with model1.fit and model2.fit). Weights and predictions do not require compile.
6

etov's answer is a great option.

But suppose you already have model1 and model2 ready and you don't want to change them, you can create the third model like this:

singleInput = Input((3,))

out1 = model1(singleInput)   
out2 = model2(singleInput)
#....
#outN = modelN(singleInput)

out = Concatenate()([out1,out2]) #[out1,out2,...,outN]
out = Dense(1, activation='sigmoid')(out)

model3 = Model(singleInput,out)

And if you already have all the models ready and don't want to change them, you can have something like this (not tested):

singleInput = Input((3,))
output = model3([singleInput,singleInput])
singleModel = Model(singleInput,output)
Improve this answer

2 Comments

This is indeed a good option, and in practice I think the two ways are nearly equivalent, with the exception of the input layer being an embedding. In that case using a common input layer vs. using a different one for each model would make a difference (both are valid - the right choice would depend on the application)
Yes. But the resulting model is not sequential.
3

Define new input layer and use model outputs directly (works in functional api):

assert model1.input_shape == model2.input_shape # make sure they got same shape

inp = tf.keras.layers.Input(shape=model1.input_shape[1:])
model = tf.keras.models.Model(inputs=[inp], outputs=[model1(inp), model2(inp)])
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.