To understand how activations propagate in wide neural networks, you must derive the mean field equations that describe the evolution of their distributions from one layer to the next. Begin by considering a fully connected feedforward network with independent and identically distributed (i.i.d.) weights and biases. Suppose the pre-activation at layer $l$ is given by $h_i^{(l)} = \sum_j W_{ij}^{(l)} x_j^{(l-1)} + b_i^{(l)}$, where $x_j^{(l-1)}$ are the post-activations from the previous layer, $W_{ij}^{(l)}$ are the weights, and $b_i^{(l)}$ are the biases. In the infinite-width limit, the Central Limit Theorem implies that each $h_i^{(l)}$ is approximately Gaussian, with a mean and variance determined by the statistics of the previous layer's activations.

Let the activations be given by $x_i^{(l)} = \phi(h_i^{(l)})$, where $\phi$ is the activation function. The mean field propagation equations then track how the distribution of $h^{(l)}$ evolves. Specifically, if the activations at layer $l-1$ are i.i.d. with zero mean and variance $q^{(l-1)}$, and the weights have variance $\sigma_w^2 / n$ and the biases variance $\sigma_b^2$, the variance of the pre-activations at layer $l$ is:

This recursive equation is the mean field propagation equation for the variance of activations. For common nonlinearities, such as the ReLU ($\phi(z) = \max(0, z)$) or hyperbolic tangent ($\phi(z) = \tanh(z)$), you can often compute the expectation analytically or numerically.

For ReLU, the propagation equation simplifies to:

since for a zero-mean Gaussian variable $z$ with variance $q$, $\mathbb{E}[\max(0, z)^2] = \frac{1}{2} q$.

For tanh, the expectation must be evaluated numerically:

These recursion relations allow you to predict how the distribution of activations changes as you move deeper into the network.

Visualizing the flow of distributions through the network, you can imagine each layer transforming the distribution of activations from the previous layer according to the mean field equation. At each step, the distribution typically remains Gaussian (for pre-activations) but its variance evolves recursively, shaped by the choice of activation function and initialization parameters. This conceptual diagram helps clarify the process:

• Input layer: distribution of input activations (often standardized);
• First hidden layer: pre-activations become Gaussian via the Central Limit Theorem, variance set by input statistics and initialization;
• Activation function: transforms the Gaussian pre-activation distribution;
• Subsequent layers: the process repeats, with each layer's pre-activation variance determined by the previous layer's output through the mean field recursion.

This flow highlights how the mean field equations serve as a map for the evolution of activation statistics throughout a deep, wide network.

The fixed-point behavior of the mean field equations is crucial for understanding the dynamics of deep networks. A fixed point occurs when the variance of activations stabilizes across layers, so that $q^{(l)} = q^{(l-1)}$. Whether or not the recursion settles to a fixed point depends on the activation function and the choice of initialization variances ($\sigma_w^2$, $\sigma_b^2$). For some activation functions and parameter choices, the variance may explode or vanish as you move deeper, leading to issues like vanishing or exploding gradients. For others, the variance converges to a stable value, ensuring healthy signal propagation. The role of the activation function is therefore central: it determines how the distribution of activations is shaped at each layer and whether stable propagation is possible. For instance, ReLU activation tends to preserve or reduce variance, while tanh can quickly squash it, depending on the initialization. Tuning the initialization to achieve fixed-point propagation is a key insight from mean field theory.

Despite its utility, the static mean field approach has important limitations. These equations describe only the propagation of distributions in untrained, randomly initialized networks, assuming infinite width and independence between neurons. They do not account for the effects of training, finite width, or correlations induced by weight updates. As such, while mean field theory provides valuable intuition about initialization and depth, it cannot capture phenomena like learning dynamics, generalization, or the breakdown of independence in practical, trained networks. Extensions and more sophisticated frameworks are necessary to address these aspects.