Backpropagation Implementation

General Approach

In forward propagation, each layer l takes the outputs from the previous layer, a^l-1, as inputs and computes its own outputs. Therefore, the forward() method of the Layer class takes the vector of previous outputs as its only parameter, while the rest of the needed information is stored within the class.

In backward propagation, each layer l only needs da^l to compute the respective gradients and return da^l-1, so the backward() method takes the da^l vector as its parameter. The rest of the required information is already stored in the Layer class.

Activation Functions Derivatives

Since derivatives of activation functions are needed for backpropagation, activation functions like ReLU and sigmoid should be structured as classes instead of standalone functions. This allows us to define both:

1. The activation function itself (implemented via the __call__() method), allowing it to be applied directly in the Layer class using self.activation(z);
2. Its derivative (implemented via the derivative() method), enabling efficient backpropagation and used in the Layer class as self.activation.derivative(z).

By structuring activation functions as objects, we can easily pass them to the Layer class and use them dynamically.

ReLu

The derivative of ReLU activation function is as follows, where z_i is an element of vector of pre-activations z:

9
1
2
3
4
5
6
class ReLU:
    def __call__(self, z):
        return np.maximum(0, z)

    def derivative(self, z):
        return (z > 0).astype(float)

Sigmoid

The derivative of sigmoid activation function is as follows:

9
1
2
3
4
5
6
7
class Sigmoid:
    def __call__(self, x):
        return 1 / (1 + np.exp(-z))

    def derivative(self, z):
        sig = self(z)
        return sig * (1 - sig)

For both of these activation functions, we apply them to the entire vector z, and the same goes for their derivatives. NumPy internally applies the operation to each element of the vector. For example, if the vector z contains 3 elements, the derivation is as follows:

The backward() Method

The backward() method is responsible for computing the gradients using the formulas below:

a^l-1 and z^l are stored as the inputs and outputs attributes in the Layer class, respectively. The activation function f is stores as the activation attribute.

Once all the required gradients are computed, the weights and biases can be updated since they are no longer needed for further computation:

Therefore, learning_rate (η) is another parameter of this method.

99
1
2
3
4
5
6
7
8
9
10
def backward(self, da, learning_rate):
    dz = ...
    d_weights = ...
    d_biases = ...
    da_prev = ...

    self.weights -= learning_rate * d_weights
    self.biases -= learning_rate * d_biases

    return da_prev