You are about to encounter one of the most fundamental loss functions in binary classification: log loss, also known as binary cross-entropy. Its mathematical definition is as follows:

Here, $y$ is the true label (0 or 1), and $\hat{p}$ is the predicted probability that the label is 1. The log loss penalizes predictions according to how much they diverge from the true label, with a particular emphasis on probabilistic confidence.

The probabilistic foundation of log loss is rooted in maximum likelihood estimation. When you predict a probability $\hat{p}$ for the label being 1, the log loss quantifies how well your prediction matches the observed outcome. If your predicted probability aligns perfectly with the true conditional probability of the label given the features, you minimize the expected log loss. This is why log loss naturally arises when fitting probabilistic classifiers: minimizing log loss is equivalent to maximizing the likelihood of the observed data under your model. Confident and correct predictions yield low log loss, while confident but incorrect predictions are heavily penalized. Uncertain predictions (where $\hat{p}$ is near 0.5) result in moderate loss regardless of the true label.