I was attempting to recreate this excerpt from the Neuromatch DL course, and here I’m training a multi-layer perceptron (in this case, with no hidden layers) on the MNIST image dataset. I noticed that every time I ran the entire code, the init loss as well as the End loss were decreasing, from (2.488, 0.0965) to (2.38,0.092) after three runs.
How is this happening? Since the MLP is being initialised again and again, shouldn’t the weights revert to a default? Why is the model fitting better?