When working with function spaces in machine learning, understanding the concepts of compactness and precompactness is crucial for analyzing the behavior of hypothesis classes and the possibility of generalization. In the context of metric and normed spaces, these properties capture important ideas about the "size" and "structure" of sets of functions.

Compactness in a metric space means that every sequence in the set has a subsequence that converges to a point within the set. More formally, a subset $K$ of a metric space $(X, d)$ is compact if every sequence $(x_n)$ in $K$ has a subsequence $(x_{n_k})$ that converges to some $x \in K$. In normed spaces, the notion is the same: a set is compact if it is sequentially compact, meaning every sequence has a convergent subsequence with the limit also in the set.

Precompactness (sometimes called relative compactness) is a slightly weaker property. A set $A$ in a metric space is precompact if its closure $\overline{A}$ is compact. Equivalently, $A$ is precompact if for every $\epsilon > 0$, $A$ can be covered by finitely many balls of radius $\epsilon$. This means that while $A$ itself might not be closed, its "limit points" are all contained in a compact set.

To make these ideas concrete, consider the closed interval $[0,1]$ in $\mathbb{R}$ with the standard metric. This set is compact: any sequence within $[0,1]$ has a convergent subsequence whose limit also lies in $[0,1]$. On the other hand, the open interval $(0,1)$ is precompact but not compact, because sequences can converge to 0 or 1, which are not included in the set, but its closure $[0,1]$ is compact.

In infinite-dimensional spaces, such as spaces of functions, compactness becomes more subtle. For example, the closed unit ball in the space of continuous functions on $[0,1]$, equipped with the sup norm, is not compact, highlighting a crucial difference from finite-dimensional settings.

A fundamental result in finite-dimensional normed spaces is the Heine-Borel theorem, which states that a subset of $\mathbb{R}^n$ is compact if and only if it is closed and bounded. This property makes working with finite-dimensional hypothesis spaces especially convenient, as compactness is easy to characterize.

However, in infinite-dimensional spaces (such as function spaces commonly used in machine learning), the Heine-Borel property fails: closed and bounded sets are generally not compact. For instance, in the space $C([0,1])$ of continuous functions on $[0,1]$ with the sup norm, the closed unit ball is closed and bounded but not compact. This distinction underlines the challenges of ensuring generalization and stability in learning when dealing with infinite-dimensional spaces, where compactness cannot be taken for granted.