The Kullback–Leibler (KL) divergence is a fundamental concept in information theory and machine learning, measuring how one probability distribution diverges from a second, expected distribution. Mathematically, it is defined as:

Here, $P$ and $Q$ are discrete probability distributions over the same set of events or outcomes, and the sum runs over all possible events $i$. In this formula, $P(i)$ represents the true probability of event $i$, while $Q(i)$ represents the probability assigned to event $i$ by the model or approximation.

KL divergence has several important properties. First, it is asymmetric:

This means the divergence from $P$ to $Q$ is not the same as from $Q$ to $P$, reflecting that the "cost" of assuming $Q$ when $P$ is true is not the same as the reverse.

Second, KL divergence is non-negative:

for all valid probability distributions $P$ and $Q$, with equality if and only if $P = Q$ everywhere.

In machine learning, KL divergence is widely used as a loss function, especially in scenarios involving probability distributions. It plays a central role in variational inference, where it measures how close an approximate distribution is to the true posterior. Additionally, KL divergence often appears as a regularization term in models that seek to prevent overfitting by encouraging distributions predicted by the model to remain close to a prior or reference distribution.