Apprendre Huber Loss: Combining MSE and MAE

The Huber loss offers a unique blend of the Mean Squared Error (MSE) and Mean Absolute Error (MAE), providing both the smoothness of $MSE$ and the robustness of $MAE$ . The formula for the Huber loss is defined piecewise, allowing it to switch between a quadratic and a linear penalty depending on the size of the prediction error. The Huber loss for a single prediction is given by:

L_\delta(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2 & \text{if } |y - \hat{y}| \leq \delta \\ \delta \cdot (|y - \hat{y}| - \frac{1}{2}\delta) & \text{otherwise} \end{cases}

Here, $y$ is the true value, $\hat{y}$ is the predicted value, and $\delta$ is a positive threshold parameter.

Note

Huber loss behaves like $MSE$ for small errors, promoting smooth optimization and sensitivity to small deviations. For large errors, it switches to $MAE$ behavior, reducing the influence of outliers and providing robustness. This balance makes Huber loss especially useful in datasets with occasional large errors or outliers.

The transition parameter $\delta$ is central to how the Huber loss functions. When the absolute error is less than or equal to $\delta$ , the loss is quadratic, just like MSE. This means small errors are penalized more strongly, encouraging precise predictions. When the error exceeds $\delta$ , the loss becomes linear, similar to MAE, which prevents large errors from having an outsized impact on the optimization process. By tuning $\delta$ , you can control the trade-off between sensitivity to small errors and robustness to outliers. A smaller $\delta$ makes the loss function more like MAE, while a larger $\delta$ makes it more like MSE.