I’m using tensorflow and Keras to do transfer learning using Resnet_50. The issue I am having is that my model seems to be doing good on accuracy but my val_loss is extremely high as well as when i try to make predictitions the accuracy is very low.

Here are the relevent parts of code:

# Create an ImageDataGenerator with data augmentation for training
data_generator = keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255,
    validation_split=0.5,
    rotation_range=30,  # Random rotations
    width_shift_range=0.2,  # Horizontal shifts
    height_shift_range=0.2,  # Vertical shifts
    shear_range=0.2,  # Shear transformations
    zoom_range=0.2,  # Zoom
    horizontal_flip=True,  # Horizontal flips
)

# Load and preprocess training data
train_data_flow = data_generator.flow_from_directory(
    dataset_path,
    target_size=(224, 224),  # Resize images to 224x224
    batch_size=32,
    class_mode='categorical',
    subset='training'  # Use training subset
)

# Load and preprocess validation data
val_data_flow = data_generator.flow_from_directory(
    dataset_path,
    target_size=(224, 224),  # Resize images to 224x224
    batch_size=32,
    class_mode='categorical',
    subset='validation'  # Use validation subset
)

Loading The Model

# Load the model from TensorFlow Hub
model_url = "https://tfhub.dev/tensorflow/resnet_50/feature_vector/1"
hub_layer = hub.KerasLayer(model_url, input_shape=(224, 224, 3) , trainable=False)

# Create a Sequential model with dropout and batch normalization
model = keras.Sequential([
    hub_layer,
    layers.Dropout(0.2),  # Lower dropout rate
    layers.Dense(256, activation='relu'),
    layers.BatchNormalization(),  # Batch normalization
    layers.Dropout(0.5),  # Dropout
    layers.Dense(9, activation='softmax')
])

# Build the Sequential model
model.build((None, 224, 224, 3))

# Summary of the model
model.summary()

I then compile :

# Compile the model
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

And Finally :

# Fit the model with early stopping
history = model.fit(
    train_data_flow,
    validation_data=val_data_flow,
    epochs=15,  # Number of epochs,
)

Here are the epochs:

Epoch 1/15

81/81 [==============================] - 489s 6s/step - loss: 0.3773 - accuracy: 0.8832 - val_loss: 0.7994 - val_accuracy: 0.7476
Epoch 2/15
81/81 [==============================] - 489s 6s/step - loss: 0.3316 - accuracy: 0.8980 - val_loss: 0.8229 - val_accuracy: 0.7378
Epoch 3/15
81/81 [==============================] - 488s 6s/step - loss: 0.3468 - accuracy: 0.8879 - val_loss: 0.8221 - val_accuracy: 0.7362
Epoch 4/15
81/81 [==============================] - 489s 6s/step - loss: 0.3148 - accuracy: 0.9011 - val_loss: 0.8380 - val_accuracy: 0.7362
Epoch 5/15
81/81 [==============================] - 488s 6s/step - loss: 0.3250 - accuracy: 0.8972 - val_loss: 0.7680 - val_accuracy: 0.7409
Epoch 6/15
81/81 [==============================] - 491s 6s/step - loss: 0.3100 - accuracy: 0.9026 - val_loss: 0.7220 - val_accuracy: 0.7616
Epoch 7/15
81/81 [==============================] - 491s 6s/step - loss: 0.2844 - accuracy: 0.9120 - val_loss: 0.7259 - val_accuracy: 0.7651
Epoch 8/15
81/81 [==============================] - 490s 6s/step - loss: 0.2811 - accuracy: 0.9007 - val_loss: 0.7722 - val_accuracy: 0.7511
Epoch 9/15
81/81 [==============================] - 490s 6s/step - loss: 0.2689 - accuracy: 0.9167 - val_loss: 0.7943 - val_accuracy: 0.7433
Epoch 10/15
81/81 [==============================] - ETA: 0s - loss: 0.2569 - accuracy: 0.9182

Predicting:

Model Accuracy on Test Set: 0.2021069059695669