Backward propagation, or backpropagation, is the process of determining how the loss function changes with respect to each parameter in the neural network. The goal is to adjust these parameters in a way that reduces the overall loss.

This process relies on the gradient descent algorithm, which uses the derivatives of the loss with respect to each layer’s pre-activation values (the raw outputs before applying the activation function) and propagates them backward through the network.

Since every layer contributes to the final prediction, the gradients are computed step by step:

1. Perform forward propagation to obtain the outputs;
2. Compute the derivative of the loss with respect to the output pre-activation;
3. Propagate this derivative backward through the layers using the chain rule;
4. Compute and use the gradients for weights and biases to update them during training.

Notation

To make explanation clearer, let's use the following notation:

• $W^l$ is the weight matrix of layer $l$;
• $b^l$ is the vector of biases of layer $l$;
• $z^l$ is the vector of pre-activations of layer $l$;
• $a^l$ is the vector of activation of layer $l$;

Therefore, setting $a^0$ to $x$ (the inputs), forward propagation in a perceptron with n layers can be described as the following sequence of operations:

To describe backpropagation mathematically, introducing the following notations:

• $da^l$: derivative of the loss with respect to the activations at layer $l$;
• $dz^l$: derivative of the loss with respect to the pre-activations at layer $l$ (before applying the activation function);
• $dW^l$: derivative of the loss with respect to the weights at layer $l$;
• $db^l$: derivative of the loss with respect to the biases at layer $l$.

Computing Gradients for the Output Layer

At the final layer $n$, the first step is to compute the gradient of the loss with respect to the activations of the output layer, denoted as $da^n$.

Then, using the chain rule, the gradient of the loss with respect to the pre-activations of the output layer is calculated as:

Here, $f'^n(z^n)$ represents the derivative of the activation function at layer $n$, and the symbol $\odot$ denotes element-wise multiplication.

This value indicates how sensitive the loss function is to changes in the pre-activation values of the output layer.

After computing $dz^n$, the next step is to calculate the gradients for the weights and biases:

These gradients describe how much each weight and bias in the output layer should be adjusted to reduce the loss.

Here, $(a^{n-1})^T$ is the transposed activation vector from the previous layer. If the original vector has the shape $n_{\text{neurons}} \times 1$, its transpose has the shape $1 \times n_{\text{neurons}}$.

To continue the backward propagation, the derivative of the loss with respect to the activations of the previous layer is computed as:

This expression allows the error signal to be passed backward through the network, enabling the adjustment of earlier layers during training.

Propagating Gradients to the Hidden Layers

For each hidden layer $l$ the procedure is the same. Given $da^l$:

1. Compute the derivative of the loss with respect to the pre-activations;
2. Compute the gradients for the weights and biases;
3. Compute $da^{l-1}$ to propagate the derivative backward.

This process is repeated for each preceding layer, step by step, until the input layer is reached.

Updating Weights and Biases

After computing the gradients for all layers, the weights and biases are updated using the gradient descent algorithm:

Here, $\alpha$ represents the learning rate, which controls how much the parameters are adjusted during each training step.

Here, $\alpha$ is the learning rate, a hyperparameter that determines the size of the adjustment applied to the weights and biases during each update step.