Learn Backpropagation Implementation | Neural Network from Scratch

General Approach

In forward propagation, each layer $l$ takes the outputs from the previous layer, $a^{l-1}$ , as inputs and computes its own outputs. Therefore, the forward() method of the Layer class takes the vector of previous outputs as its only parameter, while the rest of the needed information is stored within the class.

In backward propagation, each layer $l$ only needs $da^l$ to compute the respective gradients and return $da^{l-1}$ , so the backward() method takes the $da^l$ vector as its parameter. The rest of the required information is already stored in the Layer class.

Activation Function Derivatives

Since the derivatives of activation functions are required for backpropagation, activation functions such as ReLU and sigmoid should be implemented as classes rather than standalone functions. This structure makes it possible to define both components clearly:

The activation function itself — implemented using the __call__() method, so it can be applied directly in the Layer class with self.activation(z);
Its derivative — implemented using the derivative() method, allowing efficient computation during backpropagation via self.activation.derivative(z).

Representing activation functions as objects makes it easy to pass them to different layers and apply them dynamically during both forward and backward propagation.

ReLu

The derivative of ReLU activation function is as follows, where $z_i$ is an element of vector of pre-activations $z$ :

f'(z_i) = \begin{cases} 1, z_i > 0\\ 0, z_i \le 0 \end{cases}

class ReLU:
    def __call__(self, z):
        return np.maximum(0, z)

    def derivative(self, z):
        return (z > 0).astype(float)

Sigmoid

The derivative of sigmoid activation function is as follows:

f'(z_i) = f(z_i) \cdot (1 - f(z_i))

class Sigmoid:
    def __call__(self, x):
        return 1 / (1 + np.exp(-z))

    def derivative(self, z):
        sig = self(z)
        return sig * (1 - sig)

For both activation functions, the operation is applied to the entire vector $z$ , as well as to its derivative. NumPy automatically performs the computation element-wise, meaning each element of the vector is processed independently.

For example, if the vector $z$ contains three elements, the derivative is computed as:

f'(z) = f'\left( \begin{bmatrix} z_1\\ z_2\\ z_3 \end{bmatrix} \right) = \begin{bmatrix} f'(z_1)\\ f'(z_2)\\ f'(z_3) \end{bmatrix}

The backward() Method

The backward() method is responsible for computing the gradients using the formulas below:

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

a^{l-1} and $z^l$ are stored as the inputs and outputs attributes in the Layer class, respectively. The activation function $f$ is stores as the activation attribute.

Once all the required gradients are computed, the weights and biases can be updated since they are no longer needed for further computation:

\begin{aligned} W^l &= W^l - \alpha \cdot dW^l\\ b^l &= b^l - \alpha \cdot db^l \end{aligned}

Therefore, learning_rate ( $\alpha$ ) is another parameter of this method.

def backward(self, da, learning_rate):
    dz = ...
    d_weights = ...
    d_biases = ...
    da_prev = ...

    self.weights -= learning_rate * d_weights
    self.biases -= learning_rate * d_biases

    return da_prev

Note

The * operator performs element-wise multiplication, while the np.dot() function performs dot product in NumPy. The .T attribute transposes an array.

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 8

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Awesome!

Completion rate improved to 4

Swipe to show menu