Mean field theory has provided powerful insights into the behavior of neural networks, especially in the infinite-width limit. However, it is essential to recognize that mean field theory is fundamentally an asymptotic result: its predictions become exact only as the width of each layer approaches infinity. In practical deep learning, networks are always of finite width, and this introduces deviations from the idealized mean field behavior. At finite width, certain dependencies and statistical fluctuations are no longer negligible, and the independence assumptions underlying mean field theory begin to break down. This means that quantities such as the distribution of activations, gradients, and outputs can exhibit variability not captured by the mean field approximation.

To systematically address these deviations, you must consider fluctuation terms — the leading-order corrections to mean field predictions that arise at finite width. Mathematically, these fluctuations can often be characterized using central limit theorem-like arguments, where the mean field result represents the average behavior, and the fluctuation terms quantify the variance around this average. For example, if a pre-activation in a wide layer is modeled as a sum of many independent random variables (the outputs of the previous layer), then the mean field theory captures the mean, while the leading fluctuation term is typically of order $1/\sqrt{N}$, where $N$ is the layer width. These fluctuation terms can be expressed as additional stochastic processes or noise terms in the propagation equations, and their precise characterization depends on the network architecture and initialization scheme.

A conceptual way to visualize the distinction between mean field predictions and finite-width effects is to imagine a diagram with two panels. On the left, you see a smooth curve representing the mean field prediction for some network quantity (like the variance of activations across layers), which remains constant or evolves deterministically as width increases. On the right, for a finite-width network, the same quantity fluctuates around the mean field curve, showing random deviations due to the finite number of neurons. These fluctuations are small for wide networks but can become significant as the width decreases, leading to observable differences in training dynamics and generalization.

The study of finite-width corrections is an active area of research in theoretical deep learning. Open questions include how to systematically compute higher-order corrections to mean field theory, how these corrections interact with optimization and generalization, and whether there exist universal behaviors across different architectures and training regimes. Researchers are also exploring connections between finite-width effects and phenomena such as feature learning, expressivity, and the emergence of non-Gaussian statistics during training. Understanding these corrections is crucial for bridging the gap between theoretical predictions and the practical performance of real-world neural networks.

In summary, while mean field theory offers a valuable lens for understanding neural networks in the infinite-width limit, the finite-width effects — captured by fluctuation terms and higher-order corrections — play a crucial role in practical deep learning. These effects can influence training stability, generalization, and the emergence of complex behaviors not predicted by the mean field approximation. As you design and analyze neural networks, it is important to be aware of these limitations and to consider the impact of finite width when interpreting theoretical results.