Regularization is a fundamental concept in the theory and practice of learning with reproducing kernel Hilbert spaces (RKHS). In the context of RKHS, regularization refers to the addition of a penalty term to an optimization problem, which controls the complexity or smoothness of the solution. The regularization functional in RKHS is typically formulated as follows: given a loss function $L(y, f(x))$ that measures the discrepancy between observed data points $(x, y)$ and predictions by a function $f$ in an RKHS $H$, the regularized risk minimization problem seeks to find

where $\|f\|_H$ denotes the RKHS norm of $f$, and $\lambda > 0$ is a regularization parameter. The first term encourages the function to fit the data, while the second term penalizes complexity, as measured by the RKHS norm, which is closely related to smoothness.

The existence and uniqueness of solutions to regularized problems in RKHS is guaranteed under mild conditions. Specifically, consider the following theorem: If the loss function $L$ is convex in its second argument and lower semi-continuous, and the RKHS norm is strictly convex, then for any data $(x_i, y_i)$ and any $\lambda > 0$, there exists a unique function $f^*$ in $H$ that minimizes the regularized risk functional.

Proof sketch: The proof relies on the properties of Hilbert spaces:

• The strict convexity of the RKHS norm ensures uniqueness;
• The lower semi-continuity and coercivity of the regularized functional guarantee existence;
• The representer theorem, discussed previously, further implies that the minimizer can be expressed as a finite linear combination of kernel functions centered at the data points.

Regularization in RKHS serves to balance the fit to the training data and the smoothness of the solution. Geometrically, the regularization term $\|f\|_H^2$ can be interpreted as controlling the "size" of the function in the Hilbert space, penalizing functions that are too rough or oscillatory. When the regularization parameter $\lambda$ is large, the solution is forced to be smoother (smaller norm), possibly at the expense of fitting the data less closely. Conversely, a small $\lambda$ allows the function to fit the data more exactly but risks overfitting and producing a less smooth function. This trade-off is at the heart of modern machine learning methods based on kernels, where the choice of $\lambda$ and the kernel itself determines how well the learned function generalizes to new data.