I’m trying to build a neural network that trains on my own data. It should classify a given image as one of the four types of rooms: bathroom, bedroom, kitchen, or office. The dataset consists of about 3000 images of each type. And this is how the code looks:

data = tf.keras.utils.image_dataset_from_directory('data', shuffle=True, seed=1,
                                                       labels='inferred',
                                                       label_mode='categorical')
data_iterator = data.as_numpy_iterator()
batch = data_iterator.next()

data = data.map(lambda x, y: (tf.image.random_flip_left_right(x / 255), y))
data.as_numpy_iterator().next()

train_size = int(len(data) * .7)
val_size = int(len(data) * .2)
test_size = int(len(data) * .1)
train = data.take(train_size)
val = data.skip(train_size).take(val_size)
test = data.skip(train_size + val_size).take(test_size)

model = Sequential()

model.add(Conv2D(16, (3, 3), 1, activation='relu', input_shape=(256, 256, 3)))
model.add(MaxPooling2D())
model.add(Conv2D(32, (3, 3), 1, activation='relu'))
model.add(MaxPooling2D())
model.add(Conv2D(16, (3, 3), 1, activation='relu'))
model.add(MaxPooling2D())
model.add(Flatten())
model.add(Dense(256, activation='relu', kernel_regularizer=l2(0.01)))
model.add(Dense(4, activation='sigmoid'))

model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
                  loss=tf.keras.losses.CategoricalCrossentropy(),
                  metrics=['accuracy'])

logdir = 'logs'
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=logdir)
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

hist = model.fit(train, epochs=40, validation_data=val,
                     callbacks=[tensorboard_callback, early_stopping])

And these are results:
Loss

Accuracy:

As you can see, it’s getting pretty bad very qiuckly. And I don’t know why. I have cleaned my dataset from some noisy images and so on. I have changed optimizers, but they all gave the same result. Could the problem be in the structure? Maybe there is some kind of mistake I don’t see?