Thiết kế website giá rẻ

Question

I am trying to implement a basic neural network with an input layer of 1024 neurons, two hidden layers with 64 and 32 neurons respectively, and an output layer with 10 neurons, in NumPy.
The hidden layers are activated by the ReLU function while the output layer is activated by the sigmoid function.

Problem

Apparently during the training, values here are getting so large that the computer is not being able to handle them. I have initialised my training weights to very small values, and I don’t want to use further techniques like regularisation, because this is supposed to be a bare minimum implementation of a basic Neural Network, that is just enough for simple datasets like the MNIST dataset. The complete error message that I got is shown at the end of this post.

Formulas used

Note that I have added terms to the weights & biases when updating them instead of substracting, because the error term in the last layer itself is set to negative, which backpropagates causing all the weights and bias updates to effectively substract the negative gradient of the loss function from them.

Code

Here’s the code I wrote to implement the network:

import numpy as np
import pandas as pd

Setting the Data

np.random.seed(42)

# Setting the number of samples and features
n_samples = 1000
n_features = 1024
n_classes = 10

# Generating a synthetic dataset
X = np.random.rand(n_samples, n_features)
y = np.argmax(X @ weights, axis=1) # Gives categories from 0 to 9

Initialize weights and biases

Storing input weights

W = {
    1: np.random.uniform(-0.1, 0.1, (64, 1024)), # Weights for 1st hidden layer (64 x 1024)
    2: np.random.uniform(-0.1, 0.1, (32, 64)),   # Weights for 2nd hidden layer (32 x 64)
    3: np.random.uniform(-0.1, 0.1, (10, 32))    # Weights for output layer (10 x 32)
}

Storing neuron biases

b = {
    1: np.random.uniform(-1, 1, 64), # Biases for 1st hidden layer (64,)
    2: np.random.uniform(-1, 1, 32), # Biases for 2nd hidden layer (32,)
    3: np.random.uniform(-1, 1, 10)  # Biases for output layer (10,)
}

Storing activation functions

g = {
    1: lambda x: np.maximum(0, x),  # ReLU for 1st hidden layer
    2: lambda x: np.maximum(0, x),  # ReLU for 2nd hidden layer
    3: lambda x: 1 / (1 + np.exp(-x))  # Sigmoid for output layer
}

# Their derivatives will be needed during backpropagation
derivative_g = {
    1: lambda x: np.where(x > 0, 1, 0),  # Derivative of ReLU for 1st hidden layer
    2: lambda x: np.where(x > 0, 1, 0),  # Derivative of ReLU for 2nd hidden layer
    3: lambda x: x * (1 - x)  # Derivative of Sigmoid for output layer
}

Propagate Forward

For storing net input of neurons

z = {
    1: np.zeros(64),  # Activation of: 1st hidden layer
    2: np.zeros(32),  # Activation of: 2nd hidden layer
    3: np.zeros(10)   # Activation of: Output layer
}

For storing neuron activations

a = {
    0: np.zeros(1024), # Activation of: Input layer
    1: np.zeros(64),   # Activation of: 1st hidden layer
    2: np.zeros(32),   # Activation of: 2nd hidden layer
    3: np.zeros(10)    # Activation of: Output layer
}

Function for propagating forward

def propagate_forward(a, W, b):
    for i in range(1, len(W)+1):
        z[i] = W[i] @ a[i-1] + b[i] # Calculate net input
        a[i] = g[i](z[i])           # Apply activation function of the corresponding i-th layer

Propagate Backward

def derivative_mse_loss_f(y_true, y_pred):
    return y_pred - y_true

def propagate_backward(a, z, W, b, y_true, learning_rate):
    error_term = {}

    # For output layer:-
    L = len(W) # index of the output layer
    ## Finding the error term
    error_term[L] = -derivative_mse_loss_f(y_true, y_pred = a[L]) * derivative_g[L](z[L])
    ## Updating the weights and biases based on that error
    W[L] += learning_rate * np.outer(error_term[L], a[L-1])
    b[L] += learning_rate * error_term[L]

    # For hidden layer:-
    for i in reversed(range(1, L)): # i = 2, 1 for two hidden layers
        # Propagating error term
        error_term[i] = derivative_g[i](z[i]) * (W[i+1].T @ error_term[i+1])
        # Updating the weights and biases based on that error
        W[i] += learning_rate * np.outer(error_term[i], a[i-1])
        b[i] += learning_rate * error_term[i]

Train

This is the block where the error is seen from.

# Defining the number of iterations and learning rate
iterations = 1000
learning_rate = 0.01

# Training loop
for iteration in range(iterations):
    for i in range(X.shape[0]):
        # Selecting the input data and the corresponding target
        x = X[i]
        y_true = y[i]

        # Activating 1st layer by passing the input
        a[0] = x

        # Forward propagation
        propagate_forward(a, W, b)

        # Backward propagation
        propagate_backward(a, z, W, b, y_true, learning_rate)

Error (shown as result of the last block i.e. the block just above this):

C:UsersCircularCircleAppDataLocalTempipykernel_91362587078551.py:4: RuntimeWarning: overflow encountered in exp
  3: lambda x: 1 / (1 + np.exp(-x))  # Sigmoid for output layer
C:UsersCircularCircleAppDataLocalTempipykernel_9136396994306.py:3: RuntimeWarning: overflow encountered in matmul
  z[i] = W[i] @ a[i-1] + b[i] # Calculate net input
C:UsersCircularCircleAppDataLocalTempipykernel_9136458921334.py:5: RuntimeWarning: overflow encountered in multiply
  3: lambda x: x * (1 - x)  # Derivative of Sigmoid for output layer
C:UsersCircularCircleAppDataLocalProgramsPythonPython312Libsite-packagesnumpycorenumeric.py:925: RuntimeWarning: invalid value encountered in multiply
  return multiply(a.ravel()[:, newaxis], b.ravel()[newaxis, :], out)

I tried reducing the size of the network to a more common one with: input layer of 784 neurons, and 2 hidden layers with 16 neurons in each, while keeping output layer size the same. But that did not change the error message.

I also tried producing output label using several non-linear functions, but that didn’t change the error message as well.

Could there be any issue with translating the formula to code? Any help would be appreciated a lot.