Understanding how machine learning models learn from data requires a grasp of the concepts of expected risk and empirical risk. In statistical learning theory, the expected risk is defined as the average loss a model incurs across all possible data points drawn from the true, but usually unknown, data distribution. Mathematically, this is written as:

where $f$ is your model, $L$ is the loss function, $(x, y)$ represents data points and labels, and $P$ is the true data distribution. This formulation captures the ideal scenario: evaluating your model over every possible input it might encounter in the real world.

In reality, you do not have access to the entire data distribution $P$. Instead, you only have a finite dataset—your training data. To address this, you use the empirical risk, which averages the loss over just the observed data points. This is given by:

where $n$ is the number of samples in your dataset, and $(x_i, y_i)$ are the observed pairs. Empirical risk serves as a practical stand-in for expected risk, allowing you to optimize your model using the data at hand.

Minimizing empirical risk is at the heart of most machine learning algorithms. By finding model parameters that reduce the average loss over the training data, you hope to also reduce the expected risk on unseen data. However, relying solely on empirical risk can lead to overfitting: the model may fit the training data very closely, capturing noise or peculiarities specific to that dataset rather than general patterns. When this happens, the model's performance on new, unseen data (its true expected risk) may be poor, even though the empirical risk is very low. This highlights a fundamental challenge in machine learning—striking a balance between fitting your data well and ensuring your model generalizes beyond it.