Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Learn Derivation of PCA Using Linear Algebra | Mathematical Foundations of PCA
Dimensionality Reduction with PCA

bookDerivation of PCA Using Linear Algebra

PCA seeks a new set of axes, called principal components, such that the projected data has maximum variance. The first principal component, denoted as w1w_{\raisebox{-0.5pt}{$1$}}, is chosen to maximize the variance of the projected data:

Var(Xw1)\mathrm{Var}(X w_1)

Subject to the constraint that βˆ₯w1βˆ₯=1\|w_{\raisebox{-0.5pt}{$1$}}\| = 1. The solution to this maximization problem is the eigenvector of the covariance matrix corresponding to the largest eigenvalue.

The optimization problem is:

max⁑wΒ wTΞ£wsubjectΒ toβˆ₯wβˆ₯=1\max_{w} \ w^T \Sigma w \quad \text{subject to} \quad \|w\| = 1

The solution is any vector ww that satisfies Ξ£w=Ξ»w\Sigma w = \lambda w, where Ξ»\lambda is the corresponding eigenvalue. In other words, ww is an eigenvector of the covariance matrix Ξ£\Sigma associated with eigenvalue Ξ»\lambda.


              
12345678910111213

            
import numpy as np

# Assume cov_matrix from earlier
X = np.array([[2.5, 2.4],
              [0.5, 0.7],
              [2.2, 2.9]])
X_centered = X - np.mean(X, axis=0)
cov_matrix = (X_centered.T @ X_centered) / X_centered.shape[0]

# Find the principal component (eigenvector with largest eigenvalue)
values, vectors = np.linalg.eig(cov_matrix)
principal_component = vectors[:, np.argmax(values)]
print("First principal component:", principal_component)
copy

This principal component is the direction along which the data has the highest variance. Projecting data onto this direction gives the most informative one-dimensional representation of the original dataset.

question mark

Which statement best describes the role of the covariance matrix in the derivation of PCA using linear algebra

Select the correct answer

Everything was clear?

How can we improve it?

Thanks for your feedback!

SectionΒ 2. ChapterΒ 3

Ask AI

expand

Ask AI

ChatGPT

Ask anything or try one of the suggested questions to begin our chat

Suggested prompts:

Can you explain why the principal component is important in PCA?

How do I interpret the values of the principal component?

What does projecting data onto the principal component mean?

Awesome!

Completion rate improved to 8.33

bookDerivation of PCA Using Linear Algebra

Swipe to show menu

PCA seeks a new set of axes, called principal components, such that the projected data has maximum variance. The first principal component, denoted as w1w_{\raisebox{-0.5pt}{$1$}}, is chosen to maximize the variance of the projected data:

Var(Xw1)\mathrm{Var}(X w_1)

Subject to the constraint that βˆ₯w1βˆ₯=1\|w_{\raisebox{-0.5pt}{$1$}}\| = 1. The solution to this maximization problem is the eigenvector of the covariance matrix corresponding to the largest eigenvalue.

The optimization problem is:

max⁑wΒ wTΞ£wsubjectΒ toβˆ₯wβˆ₯=1\max_{w} \ w^T \Sigma w \quad \text{subject to} \quad \|w\| = 1

The solution is any vector ww that satisfies Ξ£w=Ξ»w\Sigma w = \lambda w, where Ξ»\lambda is the corresponding eigenvalue. In other words, ww is an eigenvector of the covariance matrix Ξ£\Sigma associated with eigenvalue Ξ»\lambda.


              
12345678910111213

            
import numpy as np

# Assume cov_matrix from earlier
X = np.array([[2.5, 2.4],
              [0.5, 0.7],
              [2.2, 2.9]])
X_centered = X - np.mean(X, axis=0)
cov_matrix = (X_centered.T @ X_centered) / X_centered.shape[0]

# Find the principal component (eigenvector with largest eigenvalue)
values, vectors = np.linalg.eig(cov_matrix)
principal_component = vectors[:, np.argmax(values)]
print("First principal component:", principal_component)
copy

This principal component is the direction along which the data has the highest variance. Projecting data onto this direction gives the most informative one-dimensional representation of the original dataset.

question mark

Which statement best describes the role of the covariance matrix in the derivation of PCA using linear algebra

Select the correct answer

Everything was clear?

How can we improve it?

Thanks for your feedback!

SectionΒ 2. ChapterΒ 3
some-alt