In modern machine learning, especially in classification tasks, you often encounter challenges such as class imbalance and model overconfidence. To address these, specialized loss functions like focal loss and label smoothing have been developed. Focal loss is particularly effective for datasets where some classes are much less frequent than others, while label smoothing is a regularization technique that helps models generalize better by preventing them from becoming too confident in their predictions.

The focal loss modifies the standard cross-entropy loss to reduce the relative loss for well-classified examples and focus more on hard, misclassified ones. Mathematically, for a binary classification problem, the focal loss is defined as:

where:

• $p_t$ is the predicted probability for the true class;
• $\alpha_t$ is a weighting factor to balance positive and negative classes;
• $\gamma$ (gamma) is the focusing parameter that adjusts the rate at which easy examples are down-weighted.

The term $(1 - p_t)^\gamma$ acts as a modulating factor. When an example is misclassified and $p_t$ is small, this factor is near 1 and the loss is unaffected. As $p_t$ increases (the example is correctly classified with high confidence), the factor goes to zero, reducing the loss contribution from these easy examples. This mechanism encourages the model to focus on learning from hard examples that are currently being misclassified.

Label smoothing is a simple yet powerful regularization technique for classification. Normally, the target label for class $k$ is represented as a one-hot vector: 1 for the correct class and 0 for all others. With label smoothing, you modify the target so that the correct class is assigned a value slightly less than 1, and the remaining probability is distributed among the other classes. For example, with a smoothing parameter $ε$, the new target for class $k$ becomes $1 - ε$, and each incorrect class receives $ε/(K-1)$, where $K$ is the number of classes.

This approach discourages the model from becoming overly confident in its predictions, which can improve generalization and reduce susceptibility to overfitting. By making the targets less certain, label smoothing helps the model learn more robust representations and can lead to better performance on unseen data.