I am working on binary classification of background and signal images using CNN. Background image comprises of white gaussian noise 2D map with underlying weak 2 point correlation. Signal image is a background image with linearly added small localized defect(s) which is well hidden under the background fluctuation. Such defects are not visible to the naked eye but have a distinct profile. Task for now is simply if network can detect images with hidden defects.

For the task, I used ordinary CNN binary classifier model:

# I did not put all of the code since it is not relevant to my question
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(...)
        ...
        self.fc1 = nn.Linear(...)
        ...

    def forward(self, x):
        x = F.relu(self.conv1(x))
        ...
        x = F.relu(self.fc1(x))
        ...
    
    return x

The question is whether there is any advantage to use ReLU (or any non-linear activation function) after Conv2d on feature maps. Since my signal is simply adding an image on top of an other image, I was told that filter application without ReLU is sufficient to bring out the hidden signature(s) for linearly mapped images. For me, both sides are really convincing.

Of course, it is true that I can run two different networks empirically to see if they yield the same or different result. But I want to know if there is an underlying theory to explain if non-linearity on feature maps is really useful.