As you have seen in previous chapters, the Neural Tangent Kernel (NTK) theory is fundamentally built on the assumption that neural networks are infinitely wide. This infinite-width limit allows the network's output to be described by a deterministic kernel and ensures that, at initialization, the network behaves as a Gaussian process. In practice, however, all neural networks have finite width, and this introduces corrections to the NTK framework.

Finite-width corrections arise because, in real networks, the law of large numbers does not hold perfectly — fluctuations and higher-order interactions between weights become significant. Mathematically, these corrections stem from the fact that, for any finite number of neurons, the empirical kernel fluctuates around its mean, and the network's output is no longer perfectly described by the NTK alone. As width increases, these fluctuations diminish, but they never fully disappear unless width is infinite. The consequences are that the training dynamics of finite-width networks can deviate from the predictions of NTK theory; for example, the NTK itself may evolve during training, and the network may exhibit non-Gaussian behavior.

There are important phenomena in deep learning that NTK theory does not capture. One of the most significant is feature learning: in practice, neural networks adjust their internal representations during training, discovering hierarchical structures and features relevant to the task. NTK theory, in its standard form, describes a regime known as lazy training, where the network's features remain essentially fixed and only the output layer changes meaningfully. This means that NTK cannot account for the emergence of new, data-dependent features or the development of hierarchical representations, both of which are central to the empirical success of deep learning. Additionally, effects such as the formation of compositional or multi-scale representations, adaptation to data structure, and non-linear training behaviors are all outside the reach of NTK predictions.

To visualize the distinction between NTK predictions and the actual behavior of deep neural networks, consider the following diagram:

This diagram highlights how NTK theory, while powerful, describes only a subset of behaviors observed in real-world deep learning. The gap represents phenomena such as feature learning and kernel evolution during training, which are not captured by NTK.

NTK theory is valid in the regime where neural networks are extremely wide and trained with small learning rates, so that the kernel remains nearly constant and the network operates in the lazy regime. In this regime, the theory provides accurate predictions for training dynamics and generalization, but its applicability diminishes as networks become narrower or as feature learning becomes significant. Open questions in the field include:

• How to systematically account for finite-width corrections;
• How feature learning emerges outside the NTK regime;
• How to bridge the gap between kernel-based and representation-based learning theories.

Understanding these limitations and extending NTK theory to capture more realistic network behaviors remains an active area of research.