I am trying to improve my Convolutional neural network by implementing a parallel neural network (nn) that outputs a weight between 0 and 1, and assigns this weight to the associated datapoint/image. So both networks receive the same image as an input, and the weight nn outputs a weight, while the regression cnn outputs one value. The results are then concatenated and passed on to the mutual custom loss function for both networks, that multiplies the squared difference between the predicted value and target value, then multiplies this with the given weight from the parallel nn: sum((Y(:,i)-T(:,i))^2)*W(i).

The problem is that the model seems to converge towards smaller weights W(i) since this will minimize the loss, how can I fix this?

I’ve tried adding regularization term proportional to the magnitude of the weight to penalize small weights.