In the study of neural networks with very large hidden layers, you model the network's weights and activations not as fixed values, but as random variables drawn from specific probability distributions. This perspective becomes essential in the infinite-width regime, where each neuron's behavior is influenced by the collective effect of many independent random weights. Instead of tracking each individual weight, you focus on the overall statistical properties — such as means and variances — that describe the entire layer. This approach allows you to analyze the network's behavior using tools from probability theory and statistical mechanics, greatly simplifying the mathematics and providing deep insight into the network's collective dynamics.

To make this concrete, consider a fully connected neural network layer. Each weight $W_{ij}$ is typically initialized as an independent random variable, often drawn from a normal distribution with mean zero and variance scaled by the layer's width. The pre-activation $h_i$ for neuron $i$ in the next layer is then a sum over many such random variables, weighted by the previous layer's activations. In the infinite-width limit, the law of large numbers ensures that this sum behaves in a highly predictable way, allowing you to describe the distribution of pre-activations across the entire layer.

Formally, the law of large numbers states that the average of a large number of independent, identically distributed random variables converges to their expected value as the number of variables grows. In the context of neural networks, suppose you have pre-activations defined as

where $W_{ij}$ are independent random variables (weights), $x_j$ are inputs, and $n$ is the width of the previous layer. As $n$ becomes very large, the distribution of $h_i$ approaches a Gaussian distribution due to the central limit theorem, with mean and variance determined by the statistics of the weights and inputs:

This means that, for a large enough width, you can accurately describe the entire distribution of pre-activations using just a few summary statistics, regardless of the specific realization of the random weights.

To help you visualize this process, imagine a diagram where each node in a layer receives inputs from many nodes in the previous layer, each connection representing a random weight. As the number of nodes (width) increases, the sum of these random contributions at each node becomes more predictable and can be represented as a smooth, bell-shaped curve — a Gaussian distribution. This distribution then propagates forward through the network: after applying the activation function, the output distribution of one layer becomes the input distribution for the next, and so on. Each layer transforms the distribution, but in the infinite-width limit, these transformations become deterministic mappings between distributions, rather than noisy, sample-dependent processes.

The transition from finite to infinite width marks a profound change in how you understand neural networks. With finite width, each layer's output is a noisy, sample-dependent function of the random weights. As you increase the width, the randomness "averages out," and the collective behavior of the network becomes increasingly deterministic. In the strict infinite-width limit, fluctuations vanish, and the propagation of signals through the network can be described entirely by the evolution of probability distributions — removing the dependence on any particular realization of the weights. This deterministic behavior underpins the mean field theory of neural networks, providing a powerful framework for analyzing deep learning systems at scale.