I am training this small model with approximately 4000 sentences for sentiment analysis (multi-class, pos, neut, neg):

model = Sequential(name='Sentiment_Analysis_MonoLSTM')
model.add(Embedding(V+1, 500, input_length=max_length, trainable=True, name='Embedding_layer'))
model.add(Conv1D(filters=64, kernel_size=4, padding='same', activation='relu', kernel_regularizer=l1_l2(0.005, 0.01), name='Conv1D_layer')
model.add(BatchNormalization(name='BatchNormalization'))
model.add(Bidirectional(LSTM(64, return_sequences=False), name='Bidirectional_LSTM_layer_1'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='relu', kernel_regularizer=l1_l2(0.005, 0.01)))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
optimizer = Adam(lr=1e-4)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=4, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.15, patience=1, min_lr=1e-7)
batch_size = 16

model.fit(train_pad, y_train, epochs=50, batch_size=batch_size, validation_data=(val_pad, y_val), callbacks=[reduce_lr, early_stopping])

I get stuck at about 50-60% validation accuracy.

Let me know what you think! Any advice works!
????

I honestly have tried a bunch of different things to add and decrease model complexity, but nothing seems to work. There is enough data to get better performance, so I don’t know whether I should think about using GRUs instead of LSTMs.