Forward and Backward Propagation

Forward Propagation

Let's start with forward propagation. Forward propagation is the process by which information passes through the neural network from the input layer to the output layer. During forward propagation, each neuron in the network takes input, processes it (using the weights and activation functions we talked about earlier), and passes the results on to the next layer of neurons. When the information reaches the output layer, the network makes a prediction or inference based on the data it has processed.

Backward Propagation

Now let's move on to backpropagation. After the neural network has made its forward propagation prediction, we can compare that prediction with the real data and calculate the network error. Backpropagation is the process in which this error information is used to traverse the network back and adjust the weights of the neurons. Basically, we tell the network, "Here's where you went wrong, let's fix it." Based on this information, the network error is reduced and it becomes more accurate in its predictions.

Note

The neural network error can be calculated in different ways depending on the task, but it is always a floating point number.

The learning process of a neural network is the repetition of these two stages (forward and back propagation) many times. With each iteration, the network gets smarter and smarter as it learns more about the data and how to process it to make accurate predictions.

It is important to understand that this process does not end when the network reaches "perfect accuracy" or an ideal state, because such a state does not exist. Instead, training usually stops when the network reaches an acceptable level of accuracy, or when it stops improving even after many training iterations.