1. Courses
  2. Mathematics for Machine Learning
  3. Lecture 3
Lecture 3 of 1418 min3D modelFree lecture

Eigenvectors and PCA

Some directions are special to a matrix — it only stretches them, never turns them. Find these eigenvectors by hand, then use the eigenvectors of a covariance matrix to discover the directions in which data varies most, and compress data with principal component analysis.

In this lecture you will

  • Define eigenvectors and eigenvalues and recognize them geometrically
  • Explain why the eigenvectors of a covariance matrix are the directions of greatest variance
  • Carry out PCA step by step — center, covariance, eigenvectors, project
  • Use explained variance to decide how many components to keep

Most vectors change direction when a matrix is applied to them. A few special ones don’t: the matrix only stretches, shrinks or flips them. These eigenvectors — from the German eigen, “own” — reveal the natural axes of a transformation. Applied to the covariance matrix of a dataset, they reveal the natural axes of the data, which is the idea behind principal component analysis.

Eigenvectors: directions that stay put

A vector v is an eigenvector of A when

Av=λvA\mathbf{v} = \lambda \mathbf{v}

The number λ is its eigenvalue: how much the vector is stretched (λ > 1), shrunk (0 < λ < 1) or flipped (λ < 0). Drag the probe around the circle and watch the angle between v and Av — it drops to zero on the eigenvectors.

InteractiveFinding eigenvectorsDrag the teal handle; try each matrix
Loading visual…

The symmetric matrix has two perpendicular eigenvector directions. A shear has only one, and a rotation has none — every direction is turned.

Eigenvector
A non-zero vector whose direction a matrix does not change: Av = λv.
Eigenvalue
The factor λ by which a matrix stretches its eigenvector.
Covariance matrix
A symmetric matrix of the variances of each feature (diagonal) and the covariances between pairs of features.
Principal component
An eigenvector of the covariance matrix: a direction in feature space, ranked by the variance along it.
Explained variance
The share of total variance captured by a component: its eigenvalue divided by the sum of all eigenvalues.

Principal component analysis

Real datasets have many correlated features. Height and arm span move together; so do page views and session length. Principal component analysis (PCA) finds new axes aligned with the way the data actually varies. Scroll through the method, one step at a time.

Scroll storyPCA, step by stepStep 1 of 6
Loading visual…
  1. 1
    Correlated data

    Each point is one person: height and arm span, measured in standard units. The two measurements rise and fall together.

  2. 2
    Center

    Subtract the mean of each feature from every point, so the cloud sits on the origin. PCA is about spread, not position.

  3. 3
    Covariance

    The covariance matrix records each feature’s variance on the diagonal and how the two features vary together off the diagonal. The ellipse shows the shape of the spread.

  4. 4
    Eigenvectors

    The eigenvectors of the covariance matrix point along the axes of the ellipse. The first, PC1, is the direction of greatest variance; the eigenvalues say how much variance each direction holds.

  5. 5
    Rotate

    Express every point in the new coordinates — its projection onto each component. The correlation is gone: the spread now lies along one axis.

  6. 6
    Reduce

    Keep only PC1. One number per person — call it “overall size” — keeps almost all the variance. The grey lines show what is thrown away.

The same steps work in any number of dimensions. In three, a flattened cloud of points hardly uses its third direction, and projecting onto the first two components loses almost nothing.

3D modelPCA in 3DDrag to rotate
Loading 3D model…
  1. A cloud in 3DEvery point needs three numbers. The cloud is long, fairly wide and very thin — but tilted, so no single original axis shows this.
  2. CenterMoving the mean to the origin centers the cloud.
  3. First componentPC1 is the direction of greatest spread. The label shows the share of the total variance it explains.
  4. All three componentsPC2 and PC3 are perpendicular to PC1 and to each other. PC3 explains only a small share: the cloud is almost flat.
  5. Project to 2DDropping PC3 flattens every point onto the plane of PC1 and PC2. Two numbers per point now keep almost all of the variance.
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_std = StandardScaler().fit_transform(X)        # PCA is sensitive to scale — standardize first
pca = PCA(n_components=0.95)                     # keep enough components for 95% of the variance
Z = pca.fit_transform(X_std)
print(pca.n_components_, pca.explained_variance_ratio_.round(3))

PCA turned a question about data into a question about a matrix. The next lecture turns to change: derivatives and gradients, the engine of training.

Key takeaways

  • An eigenvector of a matrix keeps its direction under the transformation; its eigenvalue is how much it is stretched.
  • Symmetric matrices, such as covariance matrices, always have perpendicular eigenvectors with real eigenvalues.
  • PCA finds the eigenvectors of the data’s covariance matrix; the first principal component is the direction of greatest variance.
  • Keeping only the top components compresses data and removes noise, at the cost of the variance in the dropped directions.

Check your understanding

4 questions · answers explained

Q1For a matrix A, what makes v an eigenvector?

The defining equation is Av = λv. The matrix scales v by the eigenvalue λ but does not turn it off its line.

Q2Why does a rotation of the plane by 30° have no real eigenvectors?

Every non-zero vector is turned by 30°, so none satisfies Av = λv with a real λ.

Q3In PCA, what is the first principal component?

PCA looks for directions, not original features. The top eigenvector of the covariance matrix is the unit direction that captures the most variance.

Q4The first three components of a 50-feature dataset explain 92% of the variance. What does this suggest?

High explained variance with few components means strong correlations between features. Projecting onto the top three components compresses the data while keeping 92% of its variance.