To understand the impact of training on neural networks in the mean field regime, you need to shift your perspective from tracking individual weights to studying the evolution of entire distributions. In the infinite-width limit, a neural network's parameters — such as its weights and biases — are not just a set of numbers but are described by probability distributions. During training, especially with gradient-based methods like gradient descent, these distributions evolve over time according to rules dictated by the loss function and the network's architecture. This approach is known as distributional dynamics.

Formally, consider a neural network with a very large number of neurons per layer. Instead of focusing on the trajectory of each weight, you describe the weights at each layer by a distribution, typically denoted as $μ_t$, where $t$ refers to the training time or step. The training process then becomes the study of how $μ_t$ evolves as you apply gradient descent to minimize the loss over your data. The distributional dynamics are governed by a partial differential equation (PDE) or a transport equation that describes how the probability mass in the weight space shifts under the influence of the gradient of the loss.

The mean field limit for gradient descent is a formal statement about what happens as the width of the network tends to infinity. Under certain assumptions — such as independent initialization of weights, exchangeability, and appropriate scaling of parameters — the empirical distribution of weights converges to a deterministic distribution that evolves according to the mean field PDE. This limit allows you to replace the high-dimensional, stochastic dynamics of finite networks with a deterministic, low-dimensional description. The key assumptions typically include:

• Independent and identically distributed (i.i.d.) initialization of weights;
• Sufficiently large width so that fluctuations due to finite size vanish;
• Suitable scaling of learning rates and weights to ensure non-trivial dynamics in the limit.

When these assumptions hold, the mean field theory provides a powerful framework to analyze and predict the behavior of neural networks during training.

Intuitively, this distributional viewpoint highlights a fundamental difference between the mean field regime and the finite-width case. In a standard finite network, you might imagine each weight following its own trajectory, influenced by the gradients it receives. However, in the infinite-width regime, it is more accurate to visualize the entire cloud of weights — represented by their distribution — moving and reshaping as training progresses. Training no longer modifies individual weights in isolation; instead, it continuously deforms the overall distribution in weight space. This means that learning is captured by the flow of distributions, and the evolution of activations and outputs can be predicted by tracking these distributional changes. This perspective not only simplifies the analysis but also provides deep insights into the collective behavior of large neural networks.