Sigmoid and Tanh Activations

The sigmoid and tanh activation functions are explored, as they play a crucial role in the functioning of RNNs.

• Sigmoid activation: the sigmoid function maps input values to an output range between 0 and 1. It is commonly used in binary classification tasks, as its output can be interpreted as a probability. However, it suffers from the vanishing gradient problem when the input values are very large or very small;
• Tanh activation: the tanh function is similar to the sigmoid but maps the input values to an output range between -1 and 1. It helps center the data around zero, which can aid learning. Despite its benefits, it also suffers from the vanishing gradient problem in certain situations;
• Working of sigmoid and tanh: both functions work by squashing the input values into a bounded range. The primary difference lies in their output range: sigmoid (0 to 1) vs. tanh (-1 to 1), which affects how the network processes and updates the information.

In the next chapter, we will look at how these activation functions play a role in LSTM networks and how they help overcome some of the limitations of standard RNNs.