When you move beyond the classical regime, where the number of variables $p$ is much less than the number of observations $n$, and enter the high-dimensional setting ($p$ much greater than $n$), the foundational tools of statistics begin to fail. A central example is the sample covariance matrix, which is crucial for multivariate analysis, principal component analysis, and hypothesis testing. Mathematically, the sample covariance matrix for a dataset with $n$ observations of $p$ variables is defined as

where $X_i$ is the $i$th observation and $\bar{X}$ is the sample mean. In the classical low-dimensional setting, this matrix is typically full rank and invertible, provided $n > p$. However, in high dimensions ($p > n$), the sample covariance matrix becomes rank deficient: its rank is at most $n - 1$, so it cannot be inverted. This non-invertibility has major implications for multivariate inference, such as discriminant analysis and linear regression, which rely on the inverse of the covariance matrix. In high dimensions, the lack of invertibility means that classical procedures cannot be directly applied, and any attempt to do so leads to unstable or undefined results.

The inconsistency of classical estimators in high dimensions is a fundamental issue. Consider the sample mean and sample covariance as estimators of their population counterparts. In the classical regime, the law of large numbers ensures that the sample mean converges to the true mean as the sample size increases, and the sample covariance converges to the true covariance. However, when the dimensionality $p$ grows with the sample size $n$, especially when $p/n$ does not go to zero, this convergence can break down.

Formally, suppose you have independent observations from a multivariate normal distribution with mean vector $μ$ and covariance matrix $Σ$. As $p/n$ approaches a positive constant (or even diverges), the expected squared error of the sample mean,

remains bounded away from zero or even grows, rather than vanishing as $n$ increases. Similarly, the sample covariance matrix becomes a poor estimator: not only is it non-invertible for $p > n$, but its eigenvalues are systematically biased, and the sample eigenvectors are inconsistent estimators of the population eigenvectors. A sketch of the proof involves examining the empirical spectral distribution of the sample covariance matrix, which, in high dimensions, does not converge to a point mass at the true eigenvalues but rather spreads out, reflecting the increased variability and bias introduced by high dimensionality.

Classical hypothesis testing procedures, such as Hotelling's T-squared test or likelihood ratio tests, are built on the assumption that the estimators involved are reliable and that the sample covariance matrix is invertible. In high-dimensional settings, these tests break down dramatically. The non-invertibility of the sample covariance matrix means test statistics cannot be computed as usual. Even when modifications are made, the tests suffer from inflated Type I error rates (false positives) and a severe loss of power (failure to detect true effects). The underlying reason is that the variability of estimators increases with dimension, and the null distributions of test statistics no longer match their classical forms. This breakdown highlights the urgent need for new inferential frameworks that can handle the challenges of high dimensions, often by leveraging additional structure or constraints.

The root cause of these failures lies in the breakdown of key assumptions that underpin classical statistics. These include:

• The assumption that the sample covariance is invertible;
• The assumption that estimators are consistent as the sample size grows;
• The assumption that test statistics have well-behaved null distributions.

In high-dimensional settings, these assumptions fail because the number of variables overwhelms the amount of data, leading to overfitting, instability, and unreliable inference. To overcome these issues, it becomes necessary to impose structural constraints, such as sparsity or low-rank assumptions, on the models. These constraints reduce the effective dimensionality and allow for meaningful estimation and inference even when $p$ is much larger than $n$.