A positive definite kernel is a function $K: X \times X \to \mathbb{R}$ defined on a nonempty set $X$ such that for any finite collection of points $x_1, ..., x_n \in X$ and any real numbers $c_1, ..., c_n$, the following two properties hold:

1. Symmetry: $K(x, y) = K(y, x)$ for all $x, y \in X$;
2. Positive definiteness: $$\sum_{i=1}^n \sum_{j=1}^n c_i c_j K(x_i, x_j) \geq 0$$ for all choices of $n \in \mathbb{N}$, $x_1, ..., x_n \in X$, and $c_1, ..., c_n \in \mathbb{R}$.

These two properties ensure that the kernel behaves analogously to an inner product, but in a potentially infinite-dimensional function space. The symmetry reflects the commutativity of inner products, while the positive definiteness ensures that the associated "norm" is nonnegative, mirroring the properties of inner products in Hilbert spaces.

A fundamental proposition in the theory of reproducing kernel Hilbert spaces is that every positive definite kernel induces a unique inner product on a function space. Given a positive definite kernel $K$, one can construct a Hilbert space of functions on $X$ for which $K$ acts as a reproducing kernel. The inner product in this space is defined so that $K(x, \cdot)$ behaves like a feature vector associated to the point $x$.

Proof sketch: Consider the set of finite linear combinations of kernel sections, that is, functions of the form

for some $x_1, ..., x_n \in X$ and $c_1, ..., c_n \in \mathbb{R}$. Define an inner product between two such functions $f$ and $g$ (with $g(\cdot) = \sum_{j=1}^m d_j K(y_j, \cdot)$) by

This inner product is well-defined and positive definite because of the properties of $K$. By completing this space with respect to the induced norm, you obtain a Hilbert space, called the reproducing kernel Hilbert space ($RKHS$) associated to $K$. Geometrically, this means that the kernel allows you to embed points from $X$ into a Hilbert space in such a way that the kernel value $K(x, y)$ represents the inner product between the feature representations of $x$ and $y$. This gives a powerful geometric interpretation: kernels measure similarity via inner products in a potentially high- or infinite-dimensional space, even if you only ever compute with the kernel function itself.

Several canonical examples illustrate how positive definite kernels serve as inner products in different settings, each providing its own geometric intuition.

• Linear kernel: $K(x, y) = x^T y$ for $x, y \in \mathbb{R}^d$. This is simply the standard inner product in Euclidean space, so the associated RKHS is just the space of linear functions on $\mathbb{R}^d$. Here, the kernel directly computes the inner product between vectors;
• Polynomial kernel: $K(x, y) = (x^T y + c)^p$ for $c \geq 0$, $p \in \mathbb{N}$. This kernel corresponds to the inner product in a space of all polynomials of degree at most $p$ in the coordinates of $x$. Geometrically, it maps input vectors into a higher-dimensional space where each coordinate represents a monomial of degree up to $p$, and the kernel computes the inner product there;
• Gaussian (RBF) kernel: $K(x, y) = \exp(-\|x - y\|^2 / (2\sigma^2))$ for some $\sigma > 0$. This kernel defines an inner product in an infinite-dimensional Hilbert space. The geometric picture is that each point is mapped to a function (a "bump" centered at $x$), and the kernel computes the overlap between these bumps. The Gaussian kernel measures similarity based on distance, with closer points having larger inner products.

In each case, the kernel function encodes a notion of similarity that corresponds to an inner product in a suitable function space, which may be much higher-dimensional than the original input space.