Regularization is a fundamental concept in high-dimensional statistics, serving to encode prior structural assumptions into statistical models through mathematical mechanisms such as penalty functions and constraint sets. In high-dimensional settings, where the number of parameters can be comparable to or even exceed the number of observations, classical estimation methods often fail due to overfitting and instability. Regularization addresses these challenges by introducing additional information — known as inductive bias — into the estimation process.

Mathematically, regularization modifies the objective function used in parameter estimation by adding a penalty term that reflects a preference for certain parameter structures. For example, in the context of linear regression, the regularized estimator is often defined as the solution to an optimization problem of the form

where $L(β; X, y)$ is the loss function (such as the sum of squared residuals), $P(β)$ is the penalty function (such as the $\ell_1$ or $\ell_2$ norm), and $λ$ is a non-negative tuning parameter that controls the trade-off between data fidelity and the strength of the regularization. Alternatively, regularization can be viewed as imposing a constraint on the parameter space, for example by restricting $β$ to lie within a set defined by $\|\beta\|_q \leq t$ for some $q$ and threshold $t$. Both perspectives — penalty functions and constraint sets — express structural assumptions about the underlying model, such as sparsity or smoothness, and guide the estimator toward solutions that align with these assumptions.