Learn Derivation of Whitening Transformation | Whitening and Decorrelation

To transform a dataset so that its features are uncorrelated and each has unit variance, you use a process called whitening. Whitening is especially useful in machine learning and statistics, as many algorithms assume that input features are uncorrelated and standardized. The whitening transformation can be derived step by step using eigenvalue decomposition of the data's covariance matrix.

Suppose your data matrix $X$ is of shape $(n_samples, n_features)$ and is already mean-centered (each feature has mean zero). The first step is to compute the covariance matrix $Σ$ (sigma) of the data. This is given by:

\Sigma = \frac{1}{n} X^\top X

where $n$ is the number of samples.

Next, you perform eigenvalue decomposition on the covariance matrix $Σ$ . This means you find a matrix of eigenvectors $U$ and a diagonal matrix of eigenvalues $Λ$ (lambda) such that:

\Sigma = U \Lambda U^\top

The matrix $U$ contains the eigenvectors as columns, and $Λ$ is a diagonal matrix with the corresponding eigenvalues.

The whitening transformation seeks a matrix $W$ such that the transformed data $X_{white} = X W$ has a covariance matrix equal to the identity matrix (i.e., uncorrelated features with unit variance). The transformation matrix $W$ is derived as follows:

Rotate the data using the eigenvectors: $X U$ ;
Scale each component by the inverse square root of its eigenvalue: $X U Λ^{-1/2}$ .

Thus, the full whitening transformation is:

X_{white} = X U \Lambda^{-1/2} U^\top

Here, $Λ^{-1/2}$ means taking the reciprocal of the square root of each eigenvalue on the diagonal of $Λ$ . This operation scales each principal component to have unit variance.

In summary, the whitening transformation is constructed by first diagonalizing the covariance matrix via eigenvalue decomposition, then scaling the principal components, and finally rotating the data back to the original feature space. This ensures that the resulting features are both uncorrelated and standardized.

Definition

Whitening transforms data so that all features are uncorrelated (covariances are zero) and each has unit variance (variance equals one). This is achieved by scaling and rotating the data based on the eigenvalues and eigenvectors of its covariance matrix.


              12345678910111213141516171819202122232425262728
            
import numpy as np

# Generate a simple 2D mean-centered dataset
X = np.array([
    [2.0, 0.0],
    [0.0, 2.0],
    [-2.0, 0.0],
    [0.0, -2.0]
])

# Step 1: Compute the covariance matrix
cov = np.cov(X, rowvar=False)

# Step 2: Eigenvalue decomposition
eigvals, eigvecs = np.linalg.eigh(cov)

# Step 3: Construct the whitening matrix
# Lambda^{-1/2}: inverse square root of eigenvalues
D_inv_sqrt = np.diag(1.0 / np.sqrt(eigvals))

# Whitening matrix: eigvecs @ D_inv_sqrt @ eigvecs.T
whitening_matrix = eigvecs @ D_inv_sqrt @ eigvecs.T

# Step 4: Apply the whitening transformation
X_white = X @ whitening_matrix

print("Whitened data:\n", X_white)
print("Covariance of whitened data:\n", np.cov(X_white, rowvar=False))

Everything was clear?

Thanks for your feedback!

Section 3. Chapter 2

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Suggested prompts:

Can you explain why whitening is important in machine learning?

How does eigenvalue decomposition help in the whitening process?

Can you walk me through the steps of the whitening code example?

Awesome!

Completion rate improved to 5.26

Swipe to show menu

\Sigma = \frac{1}{n} X^\top X