As you explore the behavior of neural networks in the infinite-width regime, a central phenomenon emerges: concentration of measure. This effect describes how, as the width of each layer in a neural network grows, the outputs of the network for a given input become highly predictable and exhibit vanishing variability. In the previous chapter, you saw that at initialization, a wide neural network converges in distribution to a Gaussian process. This result is a direct consequence of concentration of measure: the randomness in the weights, when averaged over an enormous number of neurons, produces outputs that are sharply peaked around their mean, with fluctuations that diminish as width increases.

To formalize this idea, consider the concept of typicality. In the infinite-width limit, almost every realization of the network's weights will yield outputs that are extremely close to the average output predicted by the Gaussian process. Mathematically, for a neural network function $f_W(x)$ parameterized by random weights $W$, the following holds as the width $n$ tends to infinity:

• The variance of $f_W(x)$ across different initializations of $W$ decreases as $1/n$;
• The probability that $f_W(x)$ deviates significantly from its mean becomes exponentially small in $n$.

This means that, for wide networks, the output for any given input is overwhelmingly likely to be very close to the expected value. As a consequence, the function space explored by random initializations becomes highly concentrated: most networks look almost identical at initialization, and the randomness in the weights has a negligible effect on the overall behavior.

To visualize this phenomenon, imagine plotting the distribution of outputs for a fixed input as the width of the network increases. For a narrow network, the outputs are spread out, reflecting significant randomness. As the width grows, the distribution becomes sharply peaked, and in the infinite-width limit, it collapses to a single value. The following diagram illustrates this shrinking variability:

Notice how the spread of the distribution narrows as the width increases, demonstrating the concentration of outputs around the mean.

While concentration of measure provides a powerful framework for understanding wide neural networks, you must recognize its limitations. These arguments rely on the assumption of infinite or extremely large width, and on the independence of weights across neurons. In practice, real networks have finite width, and certain architectures or training procedures can introduce dependencies that break the conditions required for concentration. Moreover, during training, the weights evolve and may move away from the random initialization regime where these arguments apply. As a result, the predictions of concentration-based theory may fail in networks that are not sufficiently wide, or in scenarios where feature learning or complex correlations develop during optimization. Understanding where concentration arguments hold — and where they break down — is crucial for connecting infinite-width theory to the practical behavior of neural networks.