Recurrent neural networks (RNNs) are designed to handle sequential data by retaining information from previous inputs in their internal states. This makes them ideal for tasks like language modeling and sequence prediction.

• Sequential processing: RNN processes data step-by-step, keeping track of what has come before;
• Sentence completion: given the incomplete sentence "My favourite dish is sushi. So, my favourite cuisine is _____." the RNN processes the words one by one. After seeing "sushi", it predicts the next word as "Japanese" based on prior context;
• Memory in RNNs: at each step, the RNN updates its internal state (memory) with new information, ensuring it retains context for future steps;
• Training the RNN: RNNs are trained using backpropagation through time (BPTT), where errors are passed backward through each time step to adjust weights for better predictions.

Forward Propagation

During forward propagation, the RNN processes the input data step by step:

1. Input at time step $t$: the network receives an input $x_t$ at each time step;

2. Hidden state update: the current hidden state $h_t$ is updated based on the previous hidden state $h_{t-1}$ and the current input $x_t$ using the following formula:

   $$h_t=f(W\cdot \left[\begin{array}{c}{h}_{t-1},x_t\end{array}\right]+b)$$

   • Where:
     • $W$ is the weight matrix;
     • $b$ is the bias vector;
     • $f$ is the activation function.

3. Output generation: the output $y_t$ is generated based on the current hidden state $h_t$ using the formula:

   $$y_t=g(V\cdot h_t+c)$$

   • Where:
     • $V$ is the output weight matrix;
     • $c$ is the output bias;
     • $g$ is the activation function used at the output layer.

Backpropagation Process

Backpropagation in RNNs is crucial for updating the weights and improving the model. The process is modified to account for the sequential nature of RNNs through backpropagation through time (BPTT):

1. Error calculation: the first step in BPTT is to calculate the error at each time step. This error is typically the difference between the predicted output and the actual target;

2. Gradient calculation: in Recurrent Neural Networks, the gradients of the loss function are computed by differentiating the error with respect to network parameters and propagated backward through time from the final to the initial step, which can lead to vanishing or exploding gradients, particularly in long sequences;

3. Weight update: once the gradients are computed, the weights are updated using an optimization technique like stochastic gradient descent (SGD). The weights are adjusted in such a way that the error is minimized in future iterations. The formula for updating weights is:

   $$W:=W-\eta \frac{\partial \text{Loss}}{\partial W}$$

   • Where:
     • $\eta$ is the learning rate;
     • $\frac{\partial \text{Loss}}{\partial W}$ is the gradient of the loss function with respect to the weight matrix.

In summary, RNNs are powerful because they can remember and utilize past information, making them suitable for tasks that involve sequences.