When you represent documents using the bag-of-words model, each document becomes a vector of raw term counts. While this approach is simple and often effective, it has a significant limitation: some words appear frequently across almost all documents, such as "the," "is," or "and." These common terms, called stopwords, can dominate the representation and overshadow more meaningful, distinctive words. This means documents with similar distributions of common words may appear more similar than they really are, making it harder to distinguish between genuinely related texts. To address this, you need a weighting scheme that reduces the influence of frequent terms and highlights words that are more informative for distinguishing documents.

A widely used solution is TF-IDF weighting, which stands for Term Frequency-Inverse Document Frequency. This approach assigns a weight to each term in each document, reflecting both how often the term appears in the document and how rare it is across the entire collection. The intuition is that terms that are frequent in a document but rare overall are likely to be more informative for identifying that document.

The TF-IDF score for a term in a document is computed as the product of two quantities: term frequency (TF) and inverse document frequency (IDF).

• Term Frequency (TF): measures how often a term appears in a document, usually normalized to avoid bias toward longer documents. The most common formula is:

  $$\mathrm{TF}(t, d) = \frac{f_{t,d}}{\sum_{t'} f_{t',d}}$$

  where $f_{t,d}$ is the number of times term $t$ appears in document $d$.

• Inverse Document Frequency (IDF): measures how rare a term is across all documents. It is typically defined as:

  $$\mathrm{IDF}(t, D) = \log \left( \frac{N}{n_t} \right)$$

  where $N$ is the total number of documents, and $n_t$ is the number of documents containing term $t$.

The TF-IDF weight for term $t$ in document $d$ is then:

This formula ensures that terms that are frequent in a document but rare across the collection receive the highest weights, while very common terms get lower weights.

By applying TF-IDF weighting, each document vector is transformed so that its most distinctive terms stand out, while common or uninformative words have little influence. Geometrically, this changes the direction of document vectors in the high-dimensional space, making it easier to distinguish documents based on their unique content. For example, consider two scientific articles: both may contain frequent words like "the" and "research," but only one discusses photosynthesis while the other covers quantum mechanics. After TF-IDF weighting, photosynthesis and quantum will have high weights in their respective documents, pulling the vectors apart in the space and making their differences more apparent.

This adjustment improves the effectiveness of similarity measures, such as cosine similarity, because the most relevant words contribute more to the comparison. As a result, documents with genuinely related content are more likely to be recognized as similar, while those sharing only common vocabulary remain distinct.

Normalization plays a crucial role when comparing document vectors, especially after TF-IDF weighting. Without normalization, longer documents or those with higher overall term frequencies can have larger vector magnitudes, which can skew similarity calculations. By normalizing each document vector — typically to unit length — you ensure that comparisons focus on the direction (the pattern of term importance) rather than the absolute magnitude. This makes similarity measures reflect the relative importance of terms in each document, rather than just the amount of content. In practice, normalization helps level the playing field, allowing you to compare documents fairly regardless of their length or verbosity.