I’m trying to use the Keras Tuner package to do a hyperparameter search, including tuning the number of layers in my NN.
My code looks as follows:
def model_builder(hp):
inputs = Input(shape=(40,6))
x = inputs
for i in range(hp.Int('num_LSTM_layers',1,3)):
print("l:",i)
x = LSTM(hp.Int('lstm' + str(i) + '_units',
min_value=32,
max_value=256,
step=16),
return_sequences=True)(x)
for j in range(hp.Int('num_dense_layers',1,4)):
print("d:",j)
x = Dense(hp.Int('dense' + str(j) + '_units',
min_value=16,
max_value=128,
step=16),
activation=hp.Choice("dense" + str(j) + "_activation", ["relu", "selu","leaky_relu","tanh"]))(x)
output1 = Dense(1,activation='relu',name="WhiteElo")(x)
output2 = Dense(1,activation='relu',name="BlackElo")(x)
hp_learning_rate = hp.Choice('learning_rate', values=[1e-3, 1e-2])
model = keras.Model(inputs=inputs,outputs=[output1,output2])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate),
loss={'O1Err':'mae','O2Err':'mae'},
metrics={'O1Err':'mae','O2Err':'mae'})
return model
tuner = kt.Hyperband(model_builder,
objective='val_loss',
max_epochs=100,
factor=3)
stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)
save = tf.keras.callbacks.ModelCheckpoint('model.keras', save_best_only=True,mode='auto',monitor='val_loss')
tuner.search(X,(y1,y2),epochs=100,validation_split=0.2,callbacks=[stop_early])
As you can see, I’ve added a print
in the model building step to print out the number of LSTM and dense layers in my model.
However, on running the code, I get a different number of layers appearing to what is expected, e.g.,
Search: Running Trial #3
Value |Best Value So Far |Hyperparameter
1 |1 |num_LSTM_layers
240 |144 |lstm0_units
3 |1 |num_dense_layers
96 |16 |dense0_units
leaky_relu |leaky_relu |dense0_activation
0.001 |0.01 |learning_rate
144 |96 |lstm1_units
224 |192 |lstm2_units
96 |16 |dense1_units
relu |leaky_relu |dense1_activation
2 |2 |tuner/epochs
0 |0 |tuner/initial_epoch
4 |4 |tuner/bracket
0 |0 |tuner/round
l: 0
d: 0
d: 1
d: 2
As you can see, there should be 3 dense layers, but the tuner only displays the parameter for 1 dense layer. It looks like the tuner is “one step behind”, for my previous trial, I had 2 LSTMs and 2 dense layer, and if I let it keep running, the next trial will have parameters for 3 dense layers and 1 LSTM layer.
I just can’t work out what’s causing this behaviour… any thoughts or suggestions are very much appreciated.